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ABSTRACT 


The  general  principles  of  quality  assurance  of  chemical  measurements  are 
discussed.  They  may  be  classified  as  quality  control  — what  is  done  to 
control  the  quality  of  the  measurement  process,  and  quality  assessment  — 
what  is  done  to  evaluate  the  quality  of  the  data  output.  Quality  assurance 
practices  are  considered  as  a hierarchy  with  levels  progressing  from  the 
analyst,  the  laboratory,  the  project,  to  the  program.  The  activities  of  each 
level  are  different  and  depend  upon  the  ones  beneath  it.  Recommendations  are 
presented  for  developing  credible  quality  assurance  practices  at  each  level. 
An  appendix  contains  outlines  that  may  be  used  to  develop  the  various 
documents  associated  with  a quality  assurance  program. 
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PREFACE 


This  document  was  prepared  as  part  of  a cooperative  program  between  the 
National  Marine  Pollution  Program  Office  and  the  Center  for  Analytical 
Chemistry,  National  Bureau  of  Standards.  It  discusses  the  general  principles 
of  quality  assurance  of  measurements  and  contains  comprehensive  generic 
guidelines  for  all  aspects  of  quality  assurance  efforts  ranging  from  those  of 
an  individual  laboratory  to  those  of  a monitoring  program. 

In  preparing  this  document,  it  was  recognized  that  specific  guidelines  are 
not  only  virtually  impossible  but  may  be  counterproductive  for  quality  data. 
Quality  assurance  is  not  like  a software  package  that  can  be  purchased  and 
utilized  but  rather  needs  to  be  developed  for  each  individual  measurement 
situation  and  by  each  individual  laboratory  for  its  own  use.  What  might  be 
superfluous  in  one  case  could  be  inadequate  for  another  purpose.  Actually, 
the  process  of  development  of  a credible  quality  assurance  program  is 
educational  and  could  even  be  a major  benefit  in  itself. 

Most  well-managed  laboratories  will  find  that  what  they  are  now  doing  may  be 
most  of  what  is  necesary  to  be  done.  However,  most  will  benefit  by  adopting 
a more  systematic  approach  to  measurement,  by  improvement  of  their  documenta- 
tion, and  by  a more  rigorous  statistical  evaluation  of  their  data  outputs. 

The  result  will  be  measurement  data  to  which  statistically  supported  limits 
of  uncertainty  can  be  assigned,  and  this  should  be  an  overall  objective  of 
every  laboratory. 

It  is  hoped  that  these  guidelines  will  be  helpful  to  all  laboratories  as  they 
strive  to  develop  their  individual  quality  assurance  programs.  It  is 
realized  that  all  of  the  suggestions  contained  may  not  be  applicable  to  every 
laboratory  but  it  is  hoped  that  there  are  no  serious  omissions  that  would 
seriously  affect  data  quality.  Accordingly,  each  laboratory  is  urged  to 
study  the  contents  of  this  report,  to  accept  those  parts  which  are  pertinent 
to  its  own  measurement  situation  and  which  will  be  most  useful  to  it  in  its 
efforts  to  produce  sound  and  dependable  data. 

The  financial  support  of  the  National  Marine  Pollution  Program  and  the 
encouragement  of  its  Director,  Dr.  Andrew  Robertson,  during  the  preparation 
of  this  document  are  gratefully  acknowledged. 
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1 . INTRODUCTION 


The  demand  for  measurement  data  is  ever  increasing.  Decisions  need  to  be 
made  on  such  questions  as  the  suitability  of  a material  for  an  intended 
purpose,  the  quality  of  the  environment,  and  the  health  of  individuals.  The 
vast  majority  of  such  measurements  entail  chemical  analyses  in  which  some 
chemical  property  of  the  material  or  object  of  interest  is  measured. 

Dramatic  changes  have  taken  place  in  chemical  analysis  during  the  past 
several  decades.  Most  measurements  are  now  being  made  using  complex 
instruments  in  sophisticated  processes.  Many  of  the  operations  previously 
done  manually  are  now  automated.  Moreover,  measurements  made  by  several 
analysts  and/or  laboratories  often  need  to  be  interrelated  for  use  in  a 
decision  process.  Monitoring  programs  of  regional,  national,  or  interna- 
tional scope  may  be  involved.  All  of  this  places  stringent  requirements  for 
the  reliability  and  compatibility  of  the  data. 

The  quality  of  data  is  ordinarily  evaluated  on  the  basis  of  their  uncertainty 
when  compared  with  end-use  requirements.  If  data  have  consistency  and  their 
uncertainty  is  small  when  compared  to  the  requirements,  they  are  considered 
to  be  of  adequate  quality.  When  excessively  variable  or  the  level  of  uncer- 
tainty exceeds  the  needs,  the  data  may  be  said  to  be  of  low  or  inadequate 
quality.  The  evaluation  of  data  quality  is  thus  a relative  determination. 

What  is  high  quality  for  one  situation  could  be  unacceptable  in  another. 

The  quality  of  chemical  data  may  be  judged  on  the  basis  of  two  aspects:  the 

accuracy  of  identification  of  the  parameter  measured;  and,  the  numerical 
accuracy.  The  qualitative  identification  must  be  without  reasonable  doubt. 
The  analyst  must  state  precisely  what  was  measured  and  be  able  to  prove  it. 
While  empirical  measurements,  defined  by  a specific  apparatus  or  methodology 
may  be  useful  in  some  situations,  they  must  be  specified  as  such  and  used 
with  due  consideration  of  their  limitations. 

Quantitative  measurements  are  always  estimates  of  the  value  of  the  measurand 
and  involve  some  level  of  uncertainty.  The  measurements  must  be  made  so  that 
this  uncertainty  can  be  assigned  within  a stated  probability.  Without  such  an 
assignment,  no  use  can  be  made  of  the  data  in  any  logical  sense.  To  achieve 
this,  measurements  must  be  made  in  such  a way  to  provide  statistical 
predictability.  The  experience  of  metrologists  has  demonstrated  that  such  is 
achieved  best  by  a well-designed  and  consistently  implemented  quality 
assurance  program. 

The  above  discussion  concerns  the  quality  of  measurement  data,  and  indeed  the 
quality  of  individual  measured  values.  When  considered  collectively  as  a 
body  of  numerical  information  for  regulatory  use,  for  discussion,  or 
inference,  additional  quality  descriptors  may  be  useful.  Thus  the 
Environmental  Protection  Agency  proposes  [45]: 

Completeness  - A measure  of  the  amount  of  data  obtained  from  a 
measurement  process  compared  to  the  amount  that  was  expected  to  be 
obtained  under  the  conditions  of  measurement. 
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Representativeness  - The  degree  to  which  the  data  accurately  and 
precisely  represent  a characteristic  of  a population  parameter, 
variation  of  a property,  a process  characteristic,  or  an  operational 
condition. 

Comparability  - Expresses  the  confidence  with  which  one  data  set  can  be 
compared  to  another. 

While  difficult  to  quantify,  such  characteristics  need  to  be  considered  when 
evaluating  the  usefulness  of  a set  of  data  for  scientific  interpretation. 

The  production  of  data  of  adequate  quality  is  a common  commitment  of  most 
laboratories.  However,  it  is  pursued  differently: 

• in  various  organizations; 

• within  laboratories  in  the  same  organization; 

• within  groups  in  the  same  laboratory; 

• from  time-to-time  by  the  same  individual. 

It  is  understandable  why  this  is  so.  The  kind  and  amount  of  quality 
assurance  effort  is  related  to  the  kind  of  work  that  is  done,  the  knowledge, 
skill  and  expertise  of  analysts,  and  the  quality  requirements  for  the  data. 

In  many  measurement  situations,  it  is  assumed  that  measurement  uncertainty  is 
insignificant  compared  to  the  end  use  of  the  data.  In  such  cases,  the 
measurement  process  may  not  be  optimized,  and  the  uncertainty  of  the  data  is 
not  fully  evaluated.  The  adoption  and  implementation  of  a uniform  credible 
quality  assurance  policy  within  organizations  and  across  organizational 
units,  should  minimize  much  of  the  data  uncertainties  that  exist  and 
harmonize  data  outputs. 

It  should  be  recognized  that  quality  assurance  is  not  a system  of  software 
that  can  be  installed  but  must  be  developed  to  satisfy  individual  needs. 

W.  Edwards  Deming,  one  of  the  acknowledged  leaders  of  the  quality  assurance 
effort  of  industry,  has  said,  ”An  obstacle  that  ensures  disappointment  is  the 
supposition,  all  too  prevalent,  that  quality  control  is  something  you 
install....  Actually,  to  be  successful  in  any  company,  it  must  be  a learning 
process....  with  accumulation  of  knowledge  and  experience  under  competent 
tutelage”  (1^). 

With  due  appreciation  of  the  above,  one  can  profit  from  the  experience  of 
others,  while  developing  a quality  assurance  program  for  a specific  need. 

The  information  that  follows  summarizes  the  experience  and  thinking  of  the 
measurement  community  on  the  requirements  for  reliable  measurements.  It  is 
presented  with  the  premise  that  it  will  provide  guidance  to  laboratories  in 
the  development  of  unique,  individualized  quality  assurance  programs  that  at 
the  same  time,  will  have  interlaboratory  consistency. 
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2.  MEASUREMENT  AND  CALIBRATION 


Measurement  is  essentially  a comparison  process.  Units  are  defined  by 
convention  or  consensus  agreement  and  are  established  by  suitable  standards. 
Comparators  in  the  form  of  graduated  scales  or  measuring  devices  incorporat- 
ing sensors  are  used  to  compare  an  unknown  for  which  some  property  needs  to 
be  measured  with  a standard  for  which  that  property  is  believed  to  be  known 
with  sufficient  reliability. 

While  all  measurement  involves  some  kind  of  comparison,  the  measurement  may 
be  described  as  absolute  when  only  basic  standards  are  involved  and/or  are 
interrelated  by  fundamental  relationships.  When  derived  standards  are 
involved  and/or  related  by  empirical  or  ihferrential  relationships,  the  mea- 
surement may  be  described  as  comparative.  The  majority  of  modern  analytical 
measurements  are  comparative,  although  absolute  measurements  such  as 
gravimetry  and  coulometry  are  actively  used. 

In  every  measurement  situation,  several  assumptions  are  explicitly  or 
implicitly  involved.  The  first  is  that  the  property  of  interest  is 
measurable  by  the  measurement  process.  A second  assumption  is  that  suitable 
standards  are  available.  A third  assumption  is  that  the  measurement  or 
intercomparison  is  meaningful  and  can  be  reproduced  within  describable 
limits-  The  reliability  of  the  measurement  process  will  depend  upon  the 
validity  of  these  assumptions.  Judgement  of  such  will  require  a comprehen- 
sive knowledge  of  the  methodology,  of  the  matrix  measured,  and  demonstration 
of  the  stability  and  performance  characteristics  of  the  measurement  process. 

The  influence  of  the  matrix  on  the  measurement  process  always  needs 
clarification.  The  analytical  chemist  must  demonstrate  that  this  is  negligi- 
ble or  is  adequately  compensated,  else  the  measured  values  can  hardly  have 
significance.  Calibration  procedures  including  the  use  of  matrix-matching 
standards,  spiking,  and  standard-additions  may  be  employed  but  their  adequacy 
must  be  demonstrated.  Matrix  modification  can  also  be  used  in  some  cases. 
Removal  of  the  analyte  from  the  matrix  and  transferring  it  to  a more  benign 
one  can  provide  increased  sensitivity  and  minimize  matrix  effects.  In  each 
case,  the  analytical  uncertainties  introduced  must  be  understood  and  taken 
into  account  in  reporting  results. 

Too  often,  calibration  is  done  in  a perfunctory  manner.  Analysts  are  urged 
to  give  greater  attention  to  the  calibration  process  and  to  carefully  con- 
sider the  bias  and  random  error  that  it  may  introduce.  Bias  can  result  if 
the  calibrants  used  are  inappropriate  (matrix-mismatch)  or  if  their  composi- 
tions are  inaccurately  defined.  Drift  of  a measurement  system  between  cali- 
brations is  a source  of  bias,  hence  calibration  frequency  must  be  carefully 
considered.  Calibrations  are  often  considered  as  "errorless"  and  rarely 
repeated  except  at  intervals.  As  a result,  each  individual  calibration  may 
be  based  on  a very  few  (sometimes  zero)  degrees  of  freedom.  When  calibration 
uncertainty  becomes  significant,  as  in  ultra-trace  analysis,  it  may  become  a 
major  component  of  variance  of  the  measurement  system. 

2.1  SELECTION  OF  METHODOLOGY 


The  methodology  used  must  be  appropriate  and  adequate  for  the  measurements 
that  are  required.  In  order  to  decide  the  appropriateness  of  the  methodol- 
logy,  advance  information  is  needed  on  the  parameters  to  be  measured,  tht 
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level (a)  of  concentration  expected,  and  the  minimum  differences  among  samples 
that  should  be  distinguishable.  The  operational  characteristics  of  the  pro- 
posed methodology  must  be  known  and  should  equal  or  exceed  the  requirements 
for  the  data.  In  addition,  the  repeatibility  and  reproducibility  should  be 
sufficient  to  minimize  the  number  of  replicate  measurement  that  should  be 
made.  The  selectivity  should  be  adequate  or  ways  to  eliminate  interference 
from  other  constituents  of  the  sample  must  be  available. 

The  limit  of  detection  is  an  important  consideration  since  it  sets  a lower 
limit  for  qualitative  use  of  the  methodology.  The  limit  of  quantitation 
defines  the  lower  limit  where  quantitative  measurements  can  be  made  with  rea- 
sonable (±  30%)  confidence  (41).  The  limit  of  detection  must  be  suitable  in 
most  trace  situations  and  the  limit  of  quantitation  should  be  less  than  the 
levels  at  which  quantitative  data  are  required. 

In  addition  to  the  above,  it  is  desirable  that  the  methodology  is  easy  to 
use,  has  a low  unit-cost  of  operation,  provides  a rapid  rate  of  analysis,  is 
useful  over  a wide  range  of  concentrations,  and  is  relatively  insensitive 
(rugged)  to  reasonable  variations  of  its  operational  parameters. 

A first  requirement  for  a valid  method  (49)  is  that  its  operational 
characteristics  are  adequate  for  the  data  requirements.  A second  requirement 
is  that  a standard  operations  procedure  (SOP)  has  been  developed  so  that  it 
can  be  used  in  a specified  manner  by  a competent  analyst.  Both  of  these  fac- 
tors, and  especially  the  latter,  may  be  judged  by  collaboratively  testing  the 
methodology. 

It  is  obvious  that  valid  data  cannot  be  expected  unless  valid  methodology  is 
used,  but  this  does  not  guarantee  reliable  data.  The  laboratory/ analyst  must 
demonstrate  competence  and  must  use  an  appropriate  quality  assurance  program 
to  support  the  quality  of  the  data. 


3.  THE  QUALITY  ASSURANCE  CONCEPT 


Every  experimenter  knows  that  the  absolute  accuracy  of  any  measurement  cannot 
be  guaranteed.  Yet  data  are  for  use,  so  they  must  be  trustworthy.  Their 
quality  must  be  defined,  which  means  that  the  limits  of  confidence  must  be 
evaluated  and  stated.  The  objective  of  a quality  assurance  program  is  to 
provide  a statistical  basis  for  the  assignment  of  such  limits. 

Quality  assurance  consists  of  two  separate  but  related  activities,  quality 
control  and  quality  assessment.  Both  must  be  operational  and  coordinated. 

The  following  definitions  are  offered. 

Quality  assurance:  A system  of  activities  whose  purpose  is  to  provide 

to  the  producer  or  user  of  a product  or  a service  the  assurance  that  it 
meets  defined  standards  of  quality  with  a stated  level  of  confidence. 

Quality  control:  The  overall  system  of  activities  whose  purpose  is  to 

control  the  quality  of  a product  or  service  so  that  it  meets  the  needs 
of  users.  The  aim  is  to  provide  quality  that  is  satisfactory,  adequate, 
dependable,  and  economic. 

Quality  assessment:  The  overall  system  of  activities  whose  purpose  is 

to  provide  assurance  that  the  overall  quality  control  job  is  being  done 
effectively.  It  involves  a continuing  evaluation  of  performance  of  the 
production  system  and  the  products  produced. 

The  objective  of  quality  control  is  to  fine  tune  and  maintain  a measurement 
process  in  a desired  state  of  stability  and  reproducibility.  In  such,  the 
process  can  be  considered  to  have  a high  degree  of  similarity  to  an 
industrial  production  process,  with  the  ability  to  produce  objects  (in  this 
case  data)  with  a high  degree  of  reproducibility.  Once  this  is  established, 
the  precision  may  be  defined  and  any  biases  can  usually  be  identified  and 
eliminated  or  compensated,  so  that  requisite  data  quality  is  achieved. 

Quality  assessment  provides  the  evidence  that  this  has  been  accomplished. 

The  general  approach  to  quality  assurance  of  the  measurement  process 
parallels  that  for  the  industrial  production  process  and  borrows  concepts  and 
techniques  proven  trustworthy  for  the  latter,  to  the  extent  possible.  In 
industry,  the  procedure  may  be  called  statistical  product  control.  In 
metrology,  the  procedure  may  be  called  statistical  measurement  control. 

The  quality  assurance  procedure  followed  in  a typical  production  process  is 
illustrated  in  figure  1.  Experience  dictates  the  kind  and  degree  of  control 
that  must  be  achieved  and  maintained  to  obtain  products  of  desired  quality. 
Random  samples  of  the  output  are  tested  or  inspected  with  respect  to  specif- 
ications, in  order  to  release  or  accept  the  production  lot,  to  reject  it,  or 
to  take  corrective  actions  in  the  process. 

The  quality  assurance  aspects  of  a typical  measurement  process  are 
illustrated  in  figure  2.  Quality  control  procedures  are  used  to  tune  and 
maintain  the  system  in  a state  of  statistical  control  in  which  it  may  be  con- 
sidered as  capable  of  generating  an  infinite  number  of  measurements  of  any 
material,  of  which  the  data  of  the  moment  are  a typical  sample.  Quality 
assessment  procedures  are  then  used  to  evaluate  the  quality  of  the  data  that 
are  produced.  Unfortunately,  it  is  generally  impossible,  intrinsically,  t 
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evaluate  fully  the  quality  of  the  data  on  unknown  test  materials.  However, 
if  the  measurement  system  is  in  a state  of  statistical  control,  known  test 
samples  such  as  reference  materials  (RM)  may  be  measured  concurrently  and  the 
results  compared  with  the  reference  values.  Such  a comparison  can  be  used  to 
evalutate  the  performance  of  the  measurement  system  and  permit  inferences  to 
be  made  on  the  quality  of  the  data  for  test  samples. 


Figure  1.  Production  Process  Quality  Assurance 


Figure  2.  Measurement  Process  Quality  Assurance 

Quality  assurance  activities  may  be  considered  as  a hierarchy  progressing  in 
an  ascending  order  [51]  from  the  analyst  -»•  laboratory  -»•  project  ■*  program. 
Each  level  requires  different  activities  and  has  different  responsibilities 
which  will  depend  on  those  below  it.  The  analyst  has  the  closest  control  of 
the  measurement  process  and  the  responsibilites  are  largely  technical  in 
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nature.  The  laboratory  provides  facilities  and  human  resources  and  has 
supervisory  and  managerial  responsibilities.  The  success  of  a project  will 
depend  on  all  of  these  and  in  addition  will  hinge  on  the  soundness  of  the 
technical  concepts  and  planning  to  execute  a valid  experimental  approach. 

The  project  leader,  of  course,  will  have  major  responsibilities  in  this 
regard.  A program  may  be  considered  as  a collection  of  projects  which  must  be 
properly  conceived,  coordinated,  and  managed  for  which  the  program  manager 
has  prime  responsibility.  Intercalibration  of  the  data-producing  activities 
is  a major  requirement  for  such  a program.  It  is  obvious  that  the  quality 
assurance  procedures  will  be  different  for  each  level  of  activity  and  they 
should  be  specifically  designed.  Each  level  may  be  considered  as  a sub- 
contractor for  the  ones  above  it.  If  its  quality  assurance  practices  are 
appropriate  and  well  executed,  a minimal  amount  of  inspection  or  checking 
should  be  required  by  the  upper  levels,  which  then  can  devote  their  efforts 
largely  to  the  quality  assurance  activities  peculiar  to  their  specific 
responsibilities.  Each  level  must  inspire  confidence  in  the  levels  above  it 
and  provide  supporting  evidence  for  the  quality  of  its  outputs.  The  reader 
is  referred  to  reference  51  where  the  concepts  mentioned  above  are  discussed 
in  greater  detail. 
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STATISTICAL  CONTROL 


Everyone  requiring  or  producing  measured  values  would  like  them  to  be 
accurate.  Yet,  it  is  generally  recognized  that  measurement  is  a process  of 
estimation  and  that  some  degree  of  uncertainty  is  involved.  It  is  axiomatic 
that  a meaningful  measurement  is  reproducible  yet  it  is  common  experience 
that  repetitive  measurements  will  disagree  and  that  the  means  of  such,  while 
agreeing  better,  will  also  disagree.  In  fact,  too  close  agreement  often 
indicates  a rounding-off  either  by  the  measurement  process  or  in  the 
reporting  of  the  data. 

It  is  becoming  increasingly  recognized  that  limits  of  uncertainty  (error 
bars)  must  be  established  and  associated  with  every  data  point  so  that  the 
strength  and  weakness  of  every  decision  based  on  them  may  be  evident.  This 
supports  valid  interpretation  and  prevents  over-interpretation  of  data.  The 
experimenter  or  user  of  data  may  need  to  answer  such  questions  as: 

• are  differences  in  measured  values  significant? 

• how  close  would  an  additional  value  be  expected  to  agree  with  the  data 

already  obtained? 

• what  limits  of  confidence  can  be  assigned  to  a measured  value? 

• are  the  performances  of  two  laboratories/analysts  significantly 

different? 

• are  the  results  obtained  using  one  method/ instrument  significantly 

different  from  those  obtained  by  another? 

• do  estimates  of  precision  obtained  on  several  occasions  differ? 

These  and  similar  questions  cannot  be  answered  categorically  but  only  on  a 
probability  basis,  which  requires  that  the  related  measurements  must  be  in  a 
state  of  statistical  control. 

Statistical  control  means  that  the  output  of  a measurement  process  is 
predictable  by  statistical  concepts.  The  measurement  errors  have  attained  a 
state  of  randomness  so  that  predictive  statistical  models  validly  represent 
the  data.  (Biases  may  or  may  not  be  present,  but  their  identification  will 
be  facilitated,  as  well). 

The  prerequisite  for  statistical  control  is  a stable  measurement  process.  In 
such,  the  assignable  causes  of  variation  are  kept  in  control  by  the  quality 
control  program  and  the  measurement  operations  are  optimized  and  implemented 
by  standardized  procedures  so  that  chance  causes  of  random  error  are  mini- 
mized and/or  acceptable.  The  output  of  such  a system  will  be  expected  to 
have  a limiting  mean  and  a definite  distribution  from  which  statistical 
predictions  can  be  made. 

Eisenhart  has  stated,  "To  qualify  as  a measurement  process,  a measurement 
operation  must  have  attained  what  is  known  in  industrial  quality  control 
language  as  a state  of  statistical  control.  Until  a measurement  operation 
has  been  'debugged*  to  the  extent  that  it  has  attained  a state  of  statistical 
control,  it  cannot  be  regarded  in  any  logical  sense  as  measuring  anything  at 
all"  (15). 
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5.  QUALITY  CONTROL  TECHNIQUES 

Quality  control  techniques  include  all  practices  and  procedures  that  lead  to 
statistical  control  of  the  measurement  process.  The  basic  elements  of 
quality  control  are  listed  in  table  1.  The  list  proceeds  from  the  general  to 
the  specific  and  each  element  depends  on  the  ones  above  it  [52], 

Table  1.  Basic  Elements  of  Quality  Control 

Technical  Competence  of  Staff 
Suitable  Facilities  and  Equipment 
Good  Laboratory  Practices  (GLPs) 

Good  Measurement  Practices  (GMPs) 

Standard  Operations  Procedures  (SOPs) 

Protocols  for  Specific  Purposes  (PSPs) 

Inspection 

Documentation 

Training 


5.1  COMPETENCE 

The  competence  and  expertise  of  the  technical  staff  is  the  first  requirement 
for  quality  measurements.  Despite  the  increasing  sophistication  of  modern 
instrumentation,  technical  judgement,  experience,  skill,  and  even  the 
professional  attitudes  of  the  staff  are  very  important  for  reducing  and 
maintaining  measurement  variability  at  acceptable  levels.  The  laboratory 
staff  must  have  at  least  a minimum  level  of  competence  commensurate  with  the 
skill  requirements  of  the  analytical  discipline.  Only  in  the  most  repetitive 
of  measurement  situations  can  the  chemical  measurement  process  be  reduced  to 
a mechanical  routine.  It  is  common  knowledge  that  measurement  competence 
increases  as  experience  is  gained  with  most  methods,  and  that  laboratories, 
seemingly  equal  with  respect  to  facilities  and  equipment,  can  differ 
significantly  in  the  quality  of  their  data  output. 

5.2  SUITABLE  FACILITIES  AND  EQUIPMENT 

Modern  chemical  analysis  typically  requires  specialized  facilities, 
apparatus,  and  equipment,  and  success  or  failure  often  can  be  traced  to  their 
adequacy  and  mode  of  utilization.  Thus  ultra- trace  analysis  hardly  can  be 
done  without  ultra-clean  laboratories,  and  toxic  substance  analysis  may 
require  the  use  of  containment  areas.  Humidity  and  temperature  control  are 
recognized  prerequisites  for  reliable  measurements  in  some  areas  of  analysis 
and  modern  data  processing  and  management  systems  are  indispensible  for  some 
and  are  fast  becoming  so  for  most  other  analytical  techniques. 

Specialized  apparatus  and  equipment  are  rapidly  displacing  general-purpose 
equipment  so  that  certain  kinds  of  analytical  measurements  are  limited  to 
laboratories  where  such  are  available.  In  some  cases,  apparatus  dedicated  to 
a specific  analytical  problem  may  be  needed  to  minimize  contamination  and 
memory  effects. 
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Not  only  are  such  facilities  and  equipment  required,  but  they  must  be 
adequately  maintained.  Prescribed  maintenance  schedules,  calibrations,  and 
good  housekeeping  practices  must  be  followed  to  ensure  the  quality  of  the 
work  output. 

With  such  dependency  on  facilities  and  equipment,  their  use  can  engender 
unmerited  reliance  and  even  false  security  by  both  the  producer  and  user  of 
the  data.  It  is  a fair  statement  that  suitable  facilities  and  equipment  are 
necessary  but  not  sufficient  for  the  production  of  quality  data.  They  must 
be  backed  up  by  the  other  components  of  a quality  assurance  system,  specifi- 
cally designed  for  the  analytical  problems  of  concern.  The  complex  nature  of 
the  problems  that  can  be  solved  by  such  facilities  and  equipment  increases, 
rather  than  decreases,  the  need  for  quality  assurance  of  the  measurement 
process . 

5.3  GOOD  LABORATORY  PRACTICES  AND  GOOD  MEASUREMENT  PRACTICES 


Good  laboratory  practices  (GLPs)  and  good  measurement  practices  (GMPs) 
embrace  the  total  experience  of  analysts  in  the  making  of  good  measurements. 
GLPs  refer  to  the  general  practices  that  relate  to  many  if  not  all  of  the 
measurements  made  in  a laboratory.  They  are  technique-independent  and 
address  such  subjects  as  maintenance  of  facilities,  records,  sample  handling, 
reagent  control,  and  cleaning  of  laboratory  glassware.  Good  measurement 
practices  (GMPs)  are  technique  specific.  They  can  be  extensions  of  GLPs  to 
the  requirements  of  specific  measurement  techniques  or  they  may  be  completely 
independent  of  such. 

The  subjects  for  GLPs  and  GMPs  are  best  identified  and  the  contents  developed 
by  the  laboratory  personnel  on  the  basis  of  their  experience  and  that  of 
their  peers.  GLPs  and  GMPs  must  be  documented  and  formally  implemented  in  a 
laboratory  if  good  measurements  are  to  result.  Too  often,  GLPs  and  GMPs  are 
assumed  to  be  common  knowledge  and  unnecessary  subjects  for  documentation.  It 
is  true  that  many  of  the  things  done  in  a laboratory  do  not  need  to  be  for- 
malized. Indeed,  it  would  be  virtually  impossible  to  document  every  proce- 
dure and  every  step  of  an  operation.  However,  critical  operations  can  be 
identified  and  should  be  optimized.  Any  operation  or  sub-operation  that  is 
discovered  to  be  an  assignable  cause  of  significant  variance  or  bias  is  a 
prime  subject  for  optimization  and  documentation. 

5.4  STANDARD  OPERATIONS  PROCEDURES 

Standard  operations  procedures  (SOPs)  describe  the  way  specific  operations 
and  methods  are  performed.  These  include  sampling  operations,  sample  prepa- 
ration, calibrations,  measurement  procedures,  and  any  operation  that  is  done 
on  a repetitive  basis.  "Standard"  means  that  it  specifies  the  way  the  opera- 
tion is  to  be  done  on  each  occasion,  which  may  or  may  not  be  the  procedure 
developed  by  a standards-writing  organization.  However,  when  such  are  avail- 
able, laboratories  are  well  advised  to  consider  them  since  they  represent 
peer  judgement  and  provide  a basis  for  comparability  of  data  among  user 
laboratories . 

When  methods  are  repeatedly  used,  there  is  incentive  to  write  them  up  in  a 
formal  manner,  but  those  used  on  an  infrequent  basis  are  often  based  on  an 
analyst’s  "know-how".  Such  methodology  is  seldom  optimized  and  frequently  is 
used  differently  on  each  occasion  of  use. 


10 


The  practice  of  using  unwritten  methods  should  be  discouraged.  Documentation 
of  what  was  done  is  almost  always  needed.  When  the  writing  is  done  after- 
the-fact,  it  is  likely  to  be  brief  and  sketchy  and  possibly  inaccurate. 
Describing  a method  before  its  use  is  part  of  planning  and  should  be  done 
with  as  much  care  as  that  used  for  all  other  aspects  of  a chemical  analysis. 
Colleague  review  of  all  methodology  is  a minimum  requirement  to  insure  under- 
standability  and  hence  continuity  of  a measurement  process,  as  well  as  to 
receive  the  benefits  of  technical  criticism.  The  writing  need  not  be  diffi- 
cult and  time-consuming  if  a standard  format  is  used.  This  also  provides  a 
mechanism  for  screening  for  possible  omissions  in  the  descriptions  and  for 
intercompar isons  in  some  cases. 

While  the  use  of  SOPs  may  provide  a continuity  of  measurement  experience,  no 
methodology  should  be  used  blindly.  Its  appropriateness  should  be  considered 
at  each  time  of  use.  If  used  infrequently,  it  may  be  necessary  for  the  anal- 
yst to  make  sufficient  preliminary  measurements  to  validate  competence  and  to 
demonstrate  attainment  of  statistical  control  of  the  measurement  process  (^49) 
on  each  occasion. 

A recommended  pattern  for  increasing  the  inventory  of  useful  methodology  of  a 
laboratory  is  illustrated  in  figure  3*  Certain  of  its  methods  will  be 


Figure  3.  Methodology  Selection 


in  category  A and  B.  When  new  measurement  challenges  arise,  methodology  in 
categories  C,  D,  or  E may  be  useful,  but  some  developmental  work  may  be 
required  before  it  can  be  used  in  a trustworthy  manner.  If  an  SOP  is  pre- 
pared concurrent  with  the  use  of  such,  the  methodology  can  move  to  an  A or  B 
classification  for  future  use. 

5.5  PROTOCOLS  FOR  SPECIFIC  PURPOSES  (PSPs) 


PSPs  precisely  define  the  way  a specific  measurement  sequence,  project,  or 
program  is  to  be  conducted.  The  premise  in  the  development  of  a PSP  is  that 
cost-effective  and  reliable  data  require  a well-considered  experimental  plan. 
While  quality  control  is  the  primary  objective,  the  PSP  will  prescribe  the 
quality  assessment  procedures  to  be  used  as  well.  Because  of  their  dual 
nature,  the  concept,  content,  and  procedure  for  development  of  PSP’s  will  be 
discussed  elsewhere  (see  Section  15)  in  this  document. 

5.6  OTHER  QUALITY  CONTROL  TECHNIQUES 

5.6.1  Inspection 

Inspection  is  widely  used  in  industrial  operations  as  a quality  assurance 
procedure.  Because  of  its  non-quantitative  nature,  it  is  most  useful  for 
quality  control  in  the  measurement  process,  and  less  useful  for  quality 
assessment  purposes. 

Inspection  is  largely  a subjective  examination  for  malfunction  or  for  the 
detection  of  abnormal  conditions.  A good  measurement  process  usually  will 
have  several  places  where  inspection  may  be  used  beneficially  as  suggested  by 
Figure  4.  Samples  may  be  inspected  for  missing  or  conflicting  information 
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Figure  4.  Quality  Control  by  Inspection 


about  the  sampling  process  which  could  be  cause  for  rejection.  Also,  the 
samples  may  need  to  meet  requirements  for  physical  appearance  or  phase 
homogeneity,  for  example. 

The  measurement  process  can  have  performance  specifications  for  the 
instrumentation  (stability,  for  example),  the  laboratory  environment,  and 
validity  of  calibrations  before  acceptable  data  are  possible.  Data  checks 
may  be  made  for  reasonableness  and  expression  in  proper  units,  for  example. 

Inspections  do  not  control  quality  but  may  prevent  the  reporting  of  faulty  or 
questionable  data.  A good  inspection  program  may  make  it  possible  to  obtain 
additional  samples  or  remeasurements  when  required  and  thus  reduce  gaps  in 
data  that  might  otherwise  occur. 

5.6.2  Documentation 

Documentation  is  an  important  aspect  of  every  measurement  program.  Data  must 
be  both  technically  sound  and  defensible.  While  technical  soundness  is  a 
prerequisite,  inadequate  documentation  can  cast  suspicion  on  the  technical 
merits . 

Documentation  must  address  all  aspects  of  the  measurement  process:  the 

model,  the  plan,  the  methodology,  the  calibration,  the  sample,  and  the  data 
reduction.  All  of  these  need  to  be  tied  together  in  a comprehensive  report 
that  shows  the  interrelation  of  each  phase  and  summarizes  the  data,  including 
all  of  its  limitations.  Conclusions,  when  presented,  need  to  be  fully 
supported  by  documentation. 

Details  of  the  documentation  process  are  best  addressed  in  the  GLPs.  Special 
instructions  for  documentation  of  the  data  output  of  measurement  techniques 
should  be  included  in  the  GMPs.  The  documentation  requirements  for  a 
specific  project  or  monitoring  program  should  be  planned  as  carefully  as  all 
other  aspects  of  the  measurement  program  and  specified  in  the  PSPs.  In  doing 
so,  the  documentation  practices  of  all  participating  laboratories  should  be 
examined  not  only  for  adequacy  but  for  compatibility.  The  PSPs  may  need  to 
specify  the  format  in  which  the  data  are  to  be  presented.  This  may  include 
the  developement  of  special  reporting  forms. 

Documentation  and  reporting  of  data  must  be  done  accurately.  Transcription 
errors  are  a common  problem  that  can  be  virtually  eliminated  by  care  and 
attention  to  detail.  Systematic  inspection  and  periodic  review  of  notebooks 
and  similar  primary  records  are  recommended  to  ensure  the  general  quality  of 
their  contents.  Changes  or  revisions  of  notebook  entries  must  be  justified 
and  documented.  Any  changes  should  be  made  by  crossing  out  the  original 
entry  and  substituting  the  new  value.  The  person  making  the  change  should 
initial  the  entry  and  state  why  the  change  was  made.  No  erasures  of  records 
or  data  are  permissible.  A system  for  review  of  technical  reports  including 
countersigning  by  higher  authority  is  recommended.  Also,  the  staff  members 
responsible  for  the  measurements  should  sign  reports  to  attest  the  validity 
of  the  technical  contents. 
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5.6.3  Training 

Adequate  education  and  training  are  prerequisites  for  reliable  measurement 
capability.  The  latter  can  be  distinguished  from  the  former  by  the  degree  of 
specificity.  The  complexity  of  modern  chemical  analysis  has  increased  so 
much  that  undergraduate  education  is  no  longer  sufficient  for  analytical 
chemists.  Even  highly  educated  analytical  chemists  find  the  need  from  time- 
to-time  for  additional  training  due  to  advances  in  technology  and  to  the 
adoption  of  new  technology.  On-the-job  training  in  the  quality  assurance 
aspects  of  a laboratory’s  program  is  a necessity,  as  well.  New  employees 
should  be  indoctrinated  in  the  specific  quality  assurance  practices  they  must 
follow  and  refresher  courses  may  be  needed  by  the  continuing  staff.  A labo- 
ratory’s quality  assurance  manual  can  provide  a good  basis  for  quality 
assurance  training. 

All  training  should  be  direct;  no  serial  training  should  be  permitted. 


14 


6.  QUALITY  ASSESSMENT  TECHNIQUES 


Quality  assessment  techniques  consist  of  ways  by  which  the  measurement 
process  may  be  monitored  in  order  to  infer  the  quality  of  the  data  ouput. 
Table  2 lists  a number  of  approaches  that  may  be  used.  The  central  theme  is 
the  concept  of  redundancy.  Precision  can  be  evaluated  only  by  replication 
using  stable  samples  of  known  and  even  unknown  composition.  Evaluation  of 
bias  requires  the  repetitive  measurement  of  samples  of  known  properties 
although  fewer  measurements  may  be  required  when  statistical  control  of  the 
measurement  process  is  demonstrated  by  other  means.  The  quality  assessment 
techniques  listed  in  Table  2 are  classified  as  internal  or  external  according 
to  the  source  of  the  assistance  needed  to  implement  the  assessment  technique. 

Repetitive  measurement  of  the  samples  actually  tested  is  the  classical  way  to 
evaluate  precision.  However,  this  is  a time  consuming  process  if  not  care- 
fully planned  and  it  is  the  objective  of  other  quality  assessment  techniques 
to  minimize  the  number  of  such  measurements.  Duplicate  measurement  of  an 
appropriate  number  of  test  samples  often  can  provide  much  of  the  evaluation 
of  precision  that  is  needed  and  eliminate  all  questions  of  the 
appropriateness  of  the  quality  assurance  samples. 

Table  2.  Quality  Assessment  Techniques 


Internal 

Repetitive  Measurements 
Internal  Test  Samples 
Control  Charts 
Interchange  of  Operators 
Interchange  of  Equipment 
Independent  Measurements 
Definitive  Method  Measurements 
Audits 

External 

Collaborative  Tests 

Exchange  of  Samples 

External  Reference  Materials  (ERM's) 

Standard  Reference  Materials  (SRM’s) 

Audits 


Internal  test  samples  may  consist  of  internal  reference  materials  (IRM's), 
split  samples,  spiked  samples,  and  surrogates  which  are  measured  in  suitable 
test  routines  to  evaluate  the  precision  of  the  measurement  process.  Measure- 
ments of  all  such  samples  and  indeed  of  all  quality  assurance  samples  are 
best  interpreted  by  the  use  of  control  charts.  In  this  way,  a few  measure- 
ments of  the  moment  can  join  the  experience  of  the  past  to  demonstrate 
statistical  control  and  bolster  confidence  in  the  measurement  process. 

Measurement  bias  can  be  operator,  equipment,  and/or  methodology  dependent. 
Internal  approaches  that  may  be  used  to  investigate  such  bias  include  inter- 
change of  operators  and  apparatus/equipment,  measurement  of  selected  samples 
by  independent  techniques,  and  comparisons  of  measurement  results  with  those 
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obtained,  using  a definitive  method.  The  latter  approach  is  useful  when 
suitable  methodology  is  available  and  when  external  aids,  such  as  reference 
materials,  are  not  available. 

External  evidence  for  the  quality  of  the  measurement  process  is  important 
since  it  avoids  the  dangers  of  introspection.  Several  procedures  may  be  fol- 
lowed to  provide  external  evidence  of  the  quality  of  a measurement  process. 
These  can  confirm  the  internal  evaluation  of  precision  and  provide  indepen- 
dent assessment  of  any  bias  (or  lack  thereof).  Activities  include  partici- 
pation in  collaborative  test  exercises,  exchange  of  samples  with  other 
laboratories,  and  the  analysis  of  reference  materials  obtained  from  external 
sources  (ERMs).  The  National  Bureau  of  Standards  Standard  Reference  Mate- 
rials (SRMs)  [^3,^^]  are  unexcelled  as  test  materials  to  evaluate  the  mea- 
surement process  when  they  are  properly  used  [12,50].  They  have  the  distinct 
advantage  of  wide  acceptance,  hence  provide  a basis  for  intercomparisons  of 
measurement  systems  and  test  data  taken  under  diverse  conditions  and  by 
various  laboratories. 

From  the  above  discussion,  it  should  be  clear  that  every  laboratory  has 
within-house  capability  to  estimate  the  precision  of  its  measurements.  The 
evaluation  of  bias  is  difficult,  internally,  but  facilitated  by  the  use  of 
externally  available  techniques.  No  matter  what  the  source  of  information, 
any  evidences  of  mal-performance , whether  it  be  lack  of  precision  or  intoler- 
able bias,  should  be  carefully  investigated  and  appropriate  corrective 
actions  should  be  taken  as  necessary.  After  such  actions,  follow-up  measure- 
ments should  be  undertaken  to  verify  that  the  problems  have  been  eliminated. 

Assessment  of  the  accuracy  of  qualitative  identification  is  difficult.  The 
concept  of  control  charts  is  not  readily  applicable  and  certified  reference 
materials  for  qualitative  identification  are  not  available  at  detection 
levels  where  the  problem  of  identification  is  most  acute.  Ordinarily,  reli- 
ance is  based  on  the  known  selectivity  of  the  methodology  and  knowledge  of  the 
absence  of  potential  interferents.  The  knowledge  and  experience  of  the 
analyst  can  be  a key  factor  in  decisions  on  detection. 

Identification  using  independent  techniques  and/or  several  modifications  of 
the  same  technique  (e.g.,  different  columns)  is  useful  for  verifying  the  qua- 
litative identification  of  a measured  analyte.  Spot  checks  by  a reference 
laboratory  can  be  useful  both  for  verification  of  measured  analytes  and  to 
ensure  confidence  in  question  of  "detect"  and  "non-detect" . 

Audits  are  listed  as  both  internal  and  external  quality  assessment 
techniques.  Two  kinds  of  audits  may  be  used.  Systems  audits  consist  of  both 
systematic  and  spot  checks  of  equipment,  facilities,  and  procedures  for  com- 
pliance with  the  quality  assurance  plan.  Performance  audits  depend  on  the 
use  of  test  samples  to  evaluate  laboratory  proficiency.  Monitoring  networks 
and  accrediting  agencies  often  include  audits  as  part  of  their  procedures  to 
evaluate  laboratories.  Laboratory  management  is  well  advised  to  develop  its 
own  auditing  program  and  to  conduct  internal  audits  on  both  an  announced  and 
unannounced  basis,  whether  or  not  required  by  someone  else. 

The  use  of  internal  test  samples,  particularly  with  a control  chart,  is  a 
type  of  on-going  performance  audit.  Operators  should  be  encouraged  to 
maintain  their  own  control  charts  with  real-time  response  capability. 
Supervisors  and  higher  management  levels  may  submit  samples  and  keep  their 
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own  control  charts.  Monitoring  programs  should  distribute  control  samples 
and  maintain  control  charts  on  a real-time  basis  to  assure  intercalibration 
of  stations  in  a network.  The  effectiveness  of  the  quality  assurance  process 
at  any  level  will  minimize  the  need  and  amount  of  effort  required  at  higher 
levels  to  monitor  the  measurement  process. 

Systems  audits  should  be  conducted  in  a similar  manner.  Bench-level  audits 
provide  the  potential  for  consistent  implementation  of  the  quality  assurance 
program  and  again  should  minimize  the  need  for  higher  level  inspection.  While 
the  former  can  prevent  faulty  operations,  the  latter  can  usually  only  detect 
them.  The  approach  for  conducting  a quality  audit,  described  by  Gaft  and 
Richards  (18),  is  recommended  for  guidance  in  developing  such  a program. 


17 


7.  CONTROL  CHARTS 


The  development  and  use  of  appropriate  control  charts  (13)  is  one  of  the  most 
important  techniques  for  monitoring  a measurement  process.  The  most  common 
type  involves  the  scheduled  repetitive  measurement  of  a typical  stable  test 
sample  and  plotting  the  results.  Control  limits  are  established  within  which 
the  results  are  expected  to  lie  due  to  random  fluctuations  of  the  measurement 
process.  As  long  as  they  do  so,  the  system  is  said  to  be  in  a state  of  sta- 
tistical control.  When  the  results  are  plotted  on  a sequence  or  time  (pre- 
ferable) basis,  trends  may  be  identified  before  "out-of-control”  occurs. 
Similarly,  the  standard  deviation  or  range  of  a series  of  measurements  iftay  be 
plotted  as  a control  chart  and  kept  in  parallel  with  the  above  chart  to  indi- 
cate when  decrease  in  precision  occurs  and  hence  assist  in  the  identification 
of  measurement  problems. 

It  is  recommended  that  laboratories  maintain  charts  for  the  mean  (X)  and 
difference  (R)  of  duplicate  determinations.  Furthermore,  it  is  recommended 
that  both  warning  (2s)  and  control  (3s)  limits  be  utilized.  Such  charts 
should  be  developed  for  samples  typical  of  each  class  of  tests  performed  by 
the  laboratory,  as  a minimum.  As  long  as  the  control  samples  are  stable, 
knowledge  of  their  actual  composition  is  of  lesser  interest.  While  samples 
of  unknown  composition  cannot  monitor  bias,  they  can  monitor  stability  of  the 
measurement  process,  and  bias  can  be  evaluated  otherwise.  Accordingly,  some 
of  the  control  samples  used  should  be  reference  materials  to  facilitate  the 
evaluation  of  bias. 

Laboratories  are  urged  to  consider  also  the  use  of  R (duplicate)  charts 
utilizing  the  actual  samples  tested.  While  any  control  sample  can  be  ques- 
tioned as  to  its  similarity  to  test  samples,  the  duplicate  test  sample  con- 
trol chart  can  unquestionably  monitor  precision.  By  the  use  of  a number  of 
the  above  to  monitor  short  term  precision,  limited  use  of  control  samples  to 
monitor  long-term  precision,  and  occasional  use  of  control  reference  samples 
to  monitor  for  bias  (particularly  when  developing  a measurement  process),  the 
performance  of  the  measurement  process  can  be  satisfactorily  monitored. 

All  control  charts,  to  be  most  effective,  should  be  maintained  in  a real-time 
mode,  to  the  extent  possible.  Responsive  corrective  actions  then  can  be 
taken  as  necessary  and  questions  of  the  validity  of  the  data,  due  to  out-of- 
control operation,  can  be  minimized.  The  control  chart  thus  changes  from  a 
historical  record  (only  useful  if  the  system  remains  in  control)  to  an  active 
monitor  of  the  measurement  process,  and  provides  the  opportunity  for  feed- 
back process  control. 

The  question  of  the  number  of  control  charts  to  be  maintained  needs  to  be 
addressed.  A practical  answer  should  consider:  the  historical  stability  of 

the  measurement  process;  the  risk  taken  when  the  process  is  operated  out-of- 
control; and  the  variety  of  the  work  load. 

Experience  has  shown  that  5 to  10  percent  of  the  sample  load  should  consist 
of  control  samples  of  all  kinds.  More  than  this  produces  little  extra  infor- 
mation unless  the  system  is  excessively  variable.  In  such  a case,  the  causes 
should  be  sought  and  corrective  actions  taken  to  provide  a more  stable  sys- 
tem. If  less  than  the  above  percentage  is  analyzed,  too  high  a risk  factor 
may  be  involved. 
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The  kinds  of  control  samples  to  be  used  have  already  been  discussed  briefly. 
Since  they  will  be  used  to  draw  inferences  about  the  measurement  process, 
they  should  simulate  the  actual  test  samples  or  challenge  the  measurement 
process  in  the  same  way  as  the  normal  test  samples.  The  professional  judge- 
ment of  competent  analysts  is  required  in  making  decisions  as  to  the  number 
and  type  of  control  charts  and  control  samples  that  will  be  most  beneficial 
in  monitoring  the  measurement  process. 

In  addition  to  control  samples,  control  charts  of  other  kinds  may  be  found  to 
be  very  beneficial.  Examples  of  such  include;  calibration  functions; 
extraction  efficiencies;  chemical  blanks;  and  critical  factors  of  a measure- 
ment process.  When  a critical  factor  is  identified  and  monitored,  measure- 
ment instability  can  be  identified  and  corrected  before  the  measurements 
themselves  are  affected. 
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8.  SAMPLES  AND  SAMPLING 


The  term  "sample”  as  used  in  chemical  analysis  denotes  a portion  of  a 
material  selected  in  some  manner  to  represent  a larger  body  of  material.  It 
is  that  portion,  and  sometimes  a sub-portion  (sub-sample)  which  is  analyzed 
in  order  to  draw  conclusions  about  the  body  of  material.  Unfortunately, 
there  are  often  sample-related  uncertainties  that  can  vitiate  the  conclusions 
or  even  prevent  them. 

Obviously,  the  selection  of  the  sample  is  a critical  operation.  Questions  of 
what  samples  to  take,  when,  where,  and  how  to  take  them,  and  how  many  to  take 
and  analyze  are  intimately  related  to  the  model  (concept)  of  the  problem  and 
also  depend  on  knowledge  of  the  variance  of  the  sample  population  and  the 
variance  of  measurement.  These  questions  are  very  important  but  are  largely 
beyond  the  scope  of  the  present  document.  Guidance  for  various  aspects  of 
sampling  will  be  found  in  references  26,27,28,39,^1. 

Questions  concerning  the  integrity  of  samples  will  be  discussed  in  the 
following  sections. 

8.1  UNCERTAINTY  OF  THE  SAMPLE 

The  uncertainties  of  the  sample  may  be  classified  into  two  categories: 

(a)  those  due  to  the  natural  variability  of  the  sample;  (b)  those  due  to 
changes  in  the  sample  resulting  from  operations  and  conditions  to  which  it 
may  be  subjected  on  removal  from  the  population  to  which  it  belongs,  during 
the  course  of  analysis,  and  events  that  may  occur  in  the  ensuing  interval. 

The  uncertainties  of  category  (a)  ordinarily  must  be  evaluated  and  the 
measurement  plan  must  be  devised  so  that  population  variance  and  measurement 
variance  may  be  de-coupled  and  estimated.  The  results  of  measurement  should 
include  estimates  of  uncertainty  due  to  these  two  sources. 

The  uncertainties  of  category  (b)  are  related  directly  to  the  quality 
assurance  of  the  measurement  process  and  fall  into  two  groups,  those  due  to 
sources  of  random  error  and  those  of  a systematic  nature.  Even  systematic 
errors  can  have  a random  component.  For  example,  temperature  may  have  a sys- 
tematic (and  even  predictable  and  correctable)  effect  but  its  variability 
could  produce  random  uncertainties.  Factors  which  may  affect  the  sample 
include:  the  sampling  operation;  contamination;  degradation;  interaction 

with,  the  container;  mixing. 

8.2  QUALITY  CONTROL  OF  SAMPLING 

The  best  way  to  deal  with  sources  of  sampling  errors  is  to  eliminate  them  or 
compensate  for  them  to  the  extent  possible.  Often,  this  can  be  done  by  a 
well-designed  and  implemented  quality  control  program.  Just  as  in  measure- 
ment, the  use  of  GLP's,  GMP*s,  and  SOP’s  is  the  approach  to  quality  control. 
PSP's  for  measurement  programs  should  include  protocols  for  all  aspects  of 
sampling,  related  to  the  specified  measurement  program. 

In  some  cases,  the  quality  control  program  for  sampling  can  reduce  sampling 
uncertainty  to  insignificant  levels.  When  this  is  not  achieved,  estimates  of 
the  associated  variances  should  be  made  and  included  in  the  total  analytical 
uncertainty  (see  Appendix  B). 
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8.3  QUALITY  ASSESSMENT  OF  SAMPLING 


Because  there  are  no  "sampling  reference  materials",  this  approach  to  quality 
assessment  is  not  possible.  However,  field  blanks  and  known  samples  can  be 
carried  through  various  phases  of  sampling,  including  containment  and  stor- 
age, to  monitor  such  operations.  Monitoring  of  the  cleaning  operation  for 
containers  and  of  grinding  and  other  kinds  of  sample  preparation  is  possible. 
A valid  and  precisely  implemented  chain  of  custody  can  document  that  certain 
steps  in  a sample  quality  control  program  have  been  implemented. 

8.4  CHAIN-OF-CUSTODY 

The  concept  of  "chain-of-custody"  most  often  is  viewed  as  a means  for  legal 
validation  of  samples,  but  its  use  for  quality  assurance  is  equally  if  not 
more  important.  An  adequate  system  provides  both  assurance  of  identification 
of  the  samples  that  are  analyzed  and  that  all  aspects  of  quality  control 
required  for  them  have  been  observed. 

The  simplest  system  for  use  in  a chain-of-custody  involves  an  appropriate 
label  or  tag  that  can  be  attached  securely  to  the  sample  or  its  container. 
Such  identifiers  should  be  designed  to  trace  the  history  of  a sample,  includ- 
ing the  operations  and  conditions  to  which  it  has  been  subjected,  and  indi- 
cate places  of  residence  together  with  times,  dates,  and  the  responsible 
individuals.  The  complexity  of  the  system  will  depend  upon  the  critical 
nature  of  the  samples  and  the  risk  associated  with  the  uncertainty  of  any 
significant  information.  In  large  and/or  sensitive  measurement  programs,  a 
sample  custodian  with  responsibility  for  safeguarding  samples  and  ensuring 
that  the  chain-of-custody  is  unbroken  may  be  needed. 

Whether  or  not  a sample  custodian  is  required,  the  management  of  samples  can 
be  a problem,  even  in  small-scale  operations.  Often,  samples  must  be  sub- 
divided when  various  parameters  are  measured  by  different  analysts.  Such 
sub-divisions  must  be  made  carefully  to  ensure  the  identity  of  sub-portions 
and  without  sample  deterioration  or  introduction  of  contamination.  In  some 
cases,  physical  or  chemical  operations  such  as  sieving  and  blending  or  disso- 
lution and  extraction  are  performed  in  a central  location  and  the  products 
are  distributed  to  analysts.  Obviously,  these  operations  can  be  critical  to 
the  measurement  data  and  must  be  subjected  to  appropriate  quality  control. 

The  use  of  dedicated  facilities  (such  as  clean  rooms)  and  even  reserved  areas 
within  them  may  be  required  to  carry  out  sample-preparation  operations.  In 
ultra  trace  analysis,  the  hazard  of  cross-contamination  by  on-going  samples 
and  memory-effects  of  previous  samples  and  even  from  laboratory  reagents  need 
to  be  considered  and  appropriate  precautions  must  be  exercised.  Sample 
handlers  must  be  thoroughly  trained  to  perform  their  respective  duties 
effectively  and  responsibly. 


21 


9.  LEVELS  OF  QUALITY  ASSURANCE 


Formally  or  informally,  quality  assurance  is  a necessary  part  of  the 
production  of  data.  No  one  will  quarrel  with  the  concept  that  measurements 
should  be  made  with  appropriate  care  and  that  limits  of  uncertainty  are 
necessary  for  their  logical  use.  Only  the  questions  of  how  the  goals  are  to 
be  accomplished  are  debatable. 

For  complex  operations,  for  those  involving  a number  of  participants,  and  for 
numerous  repetitive  measurements,  a formal  quality  assurance  program  is  vir- 
tually a necessity  and  the  time  and  effort  devoted  to  its  development  should 
be  more  than  compensated  by  cost-effective  and  highly  reliable  analytical 
data. 

Indeed,  the  cost  of  operation  should  not  exceed  10  percent  of  the  total 
operational  effort  and  "do-overs"  and  "lost"  data  should  be  minimized.  In 
production  operations,  it  has  been  said  that  "quality  is  free"  (11). 

In  research  investigations,  an  informal  quality  assurance  program  may  be 
adequate.  However,  the  principles  of  quality  control  and  quality  assessment 
apply  and  their  implementation,  even  in  an  informal  way,  could  constitute  a 
major  portion  of  the  experimental  effort.  Accordingly,  research  programs  can 
formalize  appropriate  parts  of  their  programs  and  substantially  reduce  the 
overall  quality  assurance  effort  in  many  cases. 
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10.  QUALITY  ASSURANCE  POLICY 

The  adoption  and  enunciation  of  a quality  assurance  policy  is  a prerequisite 
for  an  effective  quality  assurance  program.  This  will  reflect  the  degree  of 
commitment  of  management  to  quality  outputs  and  the  resources  that  will  be 
made  available  for  this  purpose.  Based  on  the  policy,  a program  can  be 
developed  and  implemented.  The  policy  should  define  the  respective  respon- 
sibilities of  the  various  organizational  levels  in  implementing  the  program. 
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1 1 . PLANNING  FOR  QUALITY  ASSURANCE 


Some  degree  of  quality  assurance  is  practiced  in  most  laboratories.  Often 
this  is  most  of  what  is  needed  if  the  past  measurement  process  has  been  well 
planned.  Ordinarily,  there  is  little  opposition  by  laboratory  personnel  to 
doing  a good  Job.  However,  there  is  a natural  aversion  to  regulations  and 
requirements,  even  when  they  define  what  is  ordinarily  done.  When  regula- 
tions include  trivia  or  incredible  requirements,  they  not  only  discredit  the 
important  ones,  but  engender  contempt  for  the  entire  program.  Accordingly, 
the  quality  assurance  program  must  be  both  realistic  and  perceived  as  such. 

The  building  blocks  for  a quality  assurance  program  have  been  described 
earlier  [51]  and  are  illustrated  in  figure  5.  The  specificity  increases  in 
descending  order.  The  policy  of  a laboratory  reflects  its  basic  measurement 
philosophy  and  the  goals  and  objectives  it  desires  to  meet.  It  will  reflect 
the  type  of  service  it  strives  to  render  and  the  desires  and  needs  of  the 
clientele*  it  serves.  As  already  discussed,  the  GLPs  and  GMPs  describe  in 
rather  specific  terms  the  ways  certain  operations  will  be  conducted.  SOPs 
define  how  specific  measurements  will  be  made  and  the  PSPs  encompass  all  of 
the  above  for  specific  measurement  activities. 

A laboratory's  quality  assurance  program  should  be  internally  motivated  by 
the  general  desire  to  produce  high  quality  outputs  but  it  is  externally  moti- 
vated as  well  by  specific  requirements  for  a specific  project.  The  former  is 
thus  program  oriented  and  usually  results  in  the  development  of  a quality 
assurance  manual.  The  latter  is  problem  oriented  and  requires  the  develop- 
ment of  project  plans.  The  density  of  the  tie-lines  in  the  figure  indicates 
the  degree  of  relatedness  of  the  central  elements  to  the  ongoing  program  or 
to  a specific  project. 

A laboratory's  quality  assurance  program,  as  delineated  in  its  quality 
assurance  manual,  should  be  sufficiently  comprehensive  to  define  most  of  the 
measurement  services  which  it  can  provide.  Also,  it  may  include  PSPs  for  its 
recurring  measurement  activities.  Ordinarily,  PSPs  will  need  to  be  developed 
for  a specific  project  of  a type  not  previously  encountered,  and  especially 
for  monitoring  programs.  Even  when  PSPs  exist,  they  may  need  to  be  re- 
examined for  their  adequacy  at  each  time  of  re-use.  The  figure  suggests 
that,  when  a laboratory  has  a well-developed  quality  assurance  program,  PSP 
development  can  consist  of  the  selection  of  elements  appropriate  to  the  pro- 
ject of  concern  with  only  minor  modifications  being  required  to  adapt  them  to 
the  specific  program. 
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Figure  5.  Quality  Assurance  Perspective 


25 


12.  DEVELOPMENT  OF  A QA  PROGRAM 


The  development  of  a quality  assurance  program  should  Involve  all  levels  of 
laboratory  personnel.  The  management  must  make  the  decision  to  develop  a 
formal  program,  and  establishes  the  policy  that  will  be  followed.  In  doing 
so,  it  should  commit  appropriate  resources  for  both  its  development  and 
implementation.  Ordinarily,  management  designates  a leader  to  develop  the 
details  of  the  program.  Management  will  also  need  to  approve  the  plans  at 
various  stages  of  its  development  and  establish  a mechanism  for  oversight  and 
implementation  of  the  program. 

The  leader  may,  or  may  not  chair  a committee,  depending  on  the  size  of  the 
organization.  In  either  case,  the  leader  must  get  the  cooperation  and 
involvement  of  the  staff  in  developing  the  details  and  will  follow 
appropriate  procedures  to  obtain  concensus  approval. 

The  staff  must  provide  technical  advice  and  guidance,  particularly  in  the 
development  of  the  GLP’s,  GMP’s,  and  SOP's.  The  staff  may  actually  write 
appropriate  parts  of  these  documents  or  review  what  is  written.  Consensus 
approval  of  the  staff  is  a necessity  for  general  acceptability  and  for  the 
preclusion  of  trivial  and  unnecessary  requirements. 

Staff  involvement  will  be  found  to  be  educational  for  them  and  this  may  be 
one  of  the  most  important  features  of  the  development  of  a formal  quality 
assurance  program.  The  development  process  will  require  a careful  considera- 
tion of  the  measurement  process  as  it  is  practiced,  internally,  and  as  it  is 
in  the  state-of-the-art.  Parallelisms  and  contrasts  between  the  practices  of 
various  groups  and  measurement  techniques  within  a laboratory,  can  be  identi- 
fied, and  the  opportunity  is  provided  to  elevate  all  operations  to  the 
highest  common  factor.  When  the  quality  assurance  program  is  assembled  into 
the  form  of  a manual,  it  can  provide  future  guidance  as  well  as  documentation 
of  present  activities. 

The  process  of  development  of  the  quality  assurance  programs  need  not  be 
complicated.  The  developers  should  identify  the  goals  to  be  accomplished  and 
the  motivation  behind  its  development.  This  will  include  an  examination  of 
the  internal  policy  and  any  QA  requirements  of  external  programs  with  which 
there  is  appreciable  interaction. 

The  next  step  is  to  identify  and  document  the  quality  assurance  procedures 
already  implemented  in  the  on-going  program.  These  should  be  compared  with 
any  requirements  and  also  with  the  experience  of  others.  Generic  guidance  is 
provided  in  the  present  paper  and  in  several  of  the  articles  in  the 
bibliography  [2, 3, 6, 7, 8, 25, ^2] . 

The  consensus  approval,  already  referred  to,  is  the  next  step.  This  will 
include  concurrence  by  the  internal  participants  of  the  program,  by  manage- 
ment, and  by  any  external  group  or  organization  requiring  a formal  QA 
program,  in  the  order  mentioned. 

Implementation  of  the  quality  assurance  program  is  as  important  as  its 
development.  All  levels  of  the  laboratory  staff  are  involved  and  the  respon- 
sibilities of  each  should  be  clearly  defined  in  the  quality  assurance  program 
document.  Management  may  designate  a quality  assurance  officer  or  coordina- 
tor, to  provide  oversight  of  the  program.  The  responsibilities  of  such  an 
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officer  may  range  from  periodic  reviewing  of  the  program  and  conducting  sys- 
tems audits  to  providing  test  materials  and  conducting  internal  performance 
audits  on  a routine  and  continuing  basis.  The  review  and  approval  of  pro- 
posed work  and  development  of  the  PSPs  for  such  may  also  be  a responsibility 
of  this  person/office.  The  quality  assurance  office  should  be  independent  of 
daily  operations  and  report  directly  to  management  so  that  unbiased 
appraisals  of  operations  can  result. 

Middle  management  (the  supervisory  staff)  has  the  responsibility  for  direct 
implementation  of  the  quality  assurance  program.  Work  of  subordinates  is 
reviewed  for  its  technical  excellence  and  conformance  with  quality  assurance 
requirements.  Middle  management  provides  training  related  to  both  aspects 
and  makes  decisions  on  corrective  actions,  including  the  identification  of 
assignable  causes  for  measurement-process  deviations. 

The  individual  members  of  the  technical  staff  have  key  responsibilities  for 
the  quality  of  the  data  output  and  for  all  related  laboratory  operations.  The 
staff  must  be  adequately  trained  to  carry  out  both  their  technical  and  qua- 
lity assurance  responsibilities.  Staff  members  often  have  the  first  oppor- 
tunity to  detect  and  correct  malfunctions  of  the  measurement  system  and  must 
be  trained  to  do  this.  Both  the  technical  and  supervisory  staff  must  be  able 
to  distinguish  between  normal  random  fluctuations  and  abnormal  deviations. 
Unnecessary  over-corrections  can  often  increase  the  variance  of  the 
measurement  process  and  should  be  avoided. 

The  quality  assurance  program  should  be  strictly  enforced  yet  there  should  be 
flexibility  to  deviate  from  it  as  necessary.  If  deviations  are  necessary  too 
frequently,  the  program  may  be  defective  and  need  to  be  amended.  Deviation 
may  be  required  due  to  both  under-  and  over-specification  of  the  quality 
assurance  program  or  by  changes  in  the  measurement  program  that  were  not 
anticipated  or  are  not  covered  by  the  program  as  originally  developed.  For 
such  reasons  the  program  should  be  continually  reviewed  and  updated  as 
required.  While  flexibility  should  be  provided,  the  objective  should  be  to 
develop  a quality  assurance  document  that  can  be  followed  with  confidence  and 
for  which  deviations  would  be  exceptional  and  infrequent. 


QA  PROGRAM  DOCUMENTATION 
Should  be  clearly  written 

Should  explain  clearly  what  it  is  intended  to  accomplish 
Should  be  accessible 

Should  be  ’’user  friendly”  otherwise  it  will  be  ignored 
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13.  QUALITY  ASSURANCE  MANUAL 


The  quality  assurance  program  and  practices  of  a laboratory  should  be 
documented.  Ordinarily  they  will  take  the  form  of  a manual  which  will 
include: 

The  QA  Program  Document 
GLPs 
GMPs 
SOPs 

Implementation  Directives 


The  quality  assurance  manual  may  be  viewed  as  an  instruction  kit  that  the 
laboratory  itself  has  developed  to  guide  its  laboratory  operations  in  the 
production  of  quality  data.  It  can  provide  internal  guidance  on  policy  and 
procedures  and  serve  as  a training  manual  for  new  employees.  It  should 
satisfy  most  external  requirements,  as  well,  so  that  its  existence  will 
minimize  effort  wherever  PSP's  need  to  be  developed. 

The  quality  assurance  manual  should  be  comprehensive  but  concise,  and 
descriptive  but  brief.  Because  some  parts  will  be  general  and  others  speci- 
fic, it  will  need  to  be  well  organized  and  indexed.  QA  manuals  may  be  made 
more  useful  when  they  are  put  on  computers.  Careful  indexing  can  facilitate 
computer  retrieval  of  information  pertinent  to  a specific  measurement 
problem,  and  thus  enhance  their  usefulness. 
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14.  QA  OF  A MEASUREMENT  PROCESS 


The  general  operational  aspects  of  the  quality  assurance  of  a measurement 
process  are  shown  in  Figure  6.  Laboratory  management  is  responsible  for 
establishing  policy  and  for  managing  both  the  technical  operations  and  the 
quality  assurance  of  the  measurement  program.  The  laboratory  staff  has  the 
responsibility  for  conducting  all  operations  and  for  their  quality  control  in 
compliance  with  the  associated  GLPs,  GMPs  and  SOPs.  The  staff  provides  also 
the  major  input  to  the  development  of  the  GLPs,  GMPs,  and  SOPs.  The  proto- 
cols to  be  followed  (PSPs)  will  depend  on  the  nature  of  the  problem  to  be 
solved  and  will  encompass  appropriate  GLPs,  GMPs  and  SOPs.  Inputs  from  other 
sources  such  as  the  client,  regulations,  or  specifications  also  influence  the 
PSPs  which  will  also  contain  the  quality  assessment  procedures  that  are  to  be 
used . 

In  the  problem  solving  mode,  samples  from  the  problem  area  are  analyzed  by 
technical  operations  that  are  in  a state  of  statistical  control,  defined  by 
the  quality  control  program.  The  data  output  is  screened  for  acceptability 
as  defined  by  the  PSP  and  released  when  found  to  be  in  conformance. 

Figure  6 is  not  intended  to  be  complete  since  some  essential  details  have 
been  omitted  for  simplicity.  In  fact,  it  should  be  considered  in  connection 
with  figure  7 which  depicts  the  overall  process  of  analytical  problem 
solving.  This  latter  figure  emphasizes  the  importance  of  an  adequate  model 
and  of  appropriate  experimental  plans  in  addition  to  the  measurement  aspects 
which  constitute  the  bottom  line. 


Figure  6.  Quality  Assurance  of  a Measurement  Program 
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Initial  planning 
Subsequent  planning 
Data  flow 


Figure  7.  The  chemical  measurement  system.  A problem  is  represented  by  a 

model  which  defines  the  data  requirements  and  utilization.  Based 
on  the  model,  a measurement  program  may  be  planned  which  addresses 
sampling,  calibration,  methodology,  and  the  quality  assurance 
procedures  to  be  utilized.  The  design  and  implementation  of  the 
chemical  measurement  system  is  often  an  interactive  process  and 
every  measurement  program  should  include  a feed-back  mechanism 
which  provides  for  the  identification  of  deficiencies  and  the 
initiation  of  corrective  actions  for  all  parts  of  the  system,  as 
required . 

All  components  of  the  system  are  critically  dependent  upon  each 
other  and  are  implicitly  or  explicitly  present  in  every  analytical 
measurement  activity.  Failure  to  recognize  this  and  to  design 
the  system  properly  is  a common  defect  of  many  analytical 
programs. 
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15.  PROTOCOLS  FOR  SPECIFIC  PURPOSES  (PSP’s) 

The  PSP  is  the  most  specific  aspect  of  a quality  assurance  program.  It 
defines  explicitly  what  is  to  be  done  in  a given  measurement  situation.  The 
situation  may  be  as  small  as  a simple  analytical  service  offered  by  a labora- 
tory or  it  may  be  as  broad  as  an  international  monitoring  program.  The  scope 
of  a PSP  is  outlined  in  table  3. 

The  PSP  is  prepared  by  the  authority  responsible  (or  designee)  for  the 
measurement  operation.  This  may  be  the  laboratory  director  in  the  case  of 
work  to  be  done  wholely  within  a single  organization  or  the  principal  offi- 
cial of  a monitoring  program.  When  a PSP  is  developed  for  a particular 
recurring  problem  it  becomes  the  way  the  pertinent  data  are  to  be  obtained. 
When  data  are  obtained  for  an  external  purpose,  all  interested  parties  may 
need  to  be  involved  in  developing  and/or  approving  the  protocol.  Needless  to 
say,  input  should  be  sought  at  the  bench  level  to  make  the  protocol 
requirements  meaningful  and  available. 

The  PSP  is  both  a planning  and  an  operational  document  and  each  aspect  is  a 
worthy  reason  for  its  careful  preparation.  It  may  be  very  brief  with  refer- 
ences to  other  documents  as  possible  (and  this  is  preferable)  or  it  may  be 
very  detailed  for  major  programs  for  which  little  precedent  exists.  The 
subjects  to  be  addressed  in  the  content  are  outlined  in  the  following. 

15.1  DESIGNATION  OF  PRINCIPAL  INVESTIGATOR 

The  person (s)  with  overall  responsibility  for  the  implementation  of  the  work 
should  be  designated  together  with  the  responsibilities  that  are  delegated. 

A quality  assurance  officer  should  also  be  designated,  as  appropriate, 
together  with  the  responsibilities  thereof.  Other  subordinate  responsibili- 
ties should  be  specified  as  appropriate,  if  not  otherwise  defined,  such  as  in 
an  organizational  structure  for  example. 

15.2  DEFINITION  OF  PROBLEM 

The  problem  to  be  solved  should  be  clearly  defined.  This  will  include  a 
concise  statement  of  the  objective  and  the  scope  (limits)  of  the  work  to  be 
undertaken.  An  overview  of  the  activities  may  be  included,  as  appropriate, 
with  references  to  other  related  ones  not  included  in  the  scope  of  the 
present  project. 

15.3  SPECIFICATION  OF  THE  MODEL 

The  model  refers  to  the  concept  of  the  problem,  the  assumptions  that  are 
made,  and  the  hypotheses  that  are  to  be  tested.  The  general  data  require- 
ments are  also  set  forth  together  with  the  precision  and  accuracy  that  should 
be  achieved. 

15.il  SPECIFICATION  OF  SAMPLES 

This  section  will  include  details  as  to  the  number  and  kind  (including 
location,  time,  etc.)  of  samples  as  required  by  the  model.  The  methods  of 
sampling,  storage,  preservation,  subsampling,  preparation  for  analysis  and 
other  pertinent  information  should  be  prescribed.  The  manner  of  distribution 
and  chain-of-custody  procedures  may  need  to  be  specified  in  some  cases. 
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15.5  SPECIFICATION  OF  DATA  BASE 


This  section  will  specify  the  data  that  should  be  obtained  including  that  for 
analytical  samples  and  for  quality  control  and  quality  assessment.  Measure- 
ment sequences  may  also  be  specified.  The  specification  of  the  data  base 
required  to  qualify  a participant  (or  contractors)  in  a monitoring  program  as 
peers,  prior  to  their  participation,  is  a part  of  this  section. 

15.6  SPECIFICATION  OF  METHODOLOGY 

The  measurement  methods  to  be  used  should  be  specified  in  sufficient  detail 
so  that  no  misunderstandings  should  arise  as  to  what  is  to  be  done.  Even 
when  standard  methods  are  specified,  there  may  need  to  be  additional 
information  such  as  explanatory  notes,  for  example.  Any  GLPs  and  GMPs,  not 
given  explicitly  in  the  methods,  but  deemed  as  necessary  to  obtain  measure- 
ments of  requisite  quality,  should  be  included.  Reference  to  appropriate 
GLPs  and  GMPs  described  elsewhere  may  be  sufficient,  but  they  should  be 
readily  available  and  reviewed  for  their  applicability  and  adequacy  in  the 
present  situation.  Likewise,  where  deviations  or  exceptions  from  standard 
methodology  are  permitted,  they  should  be  indicated  clearly. 

15.7  SPECIFICATION  OF  CONTROLS 

This  section  will  describe  all  activities  that  will  be  undertaken  to  assure 
the  quality  of  the  data.  Any  necessary  preliminary  activities  such  as 
instrument  warm  up  and  performance  checks  should  be  described  if  not  included 
in  the  SOP's.  The  corrective  actions  to  be  taken  and  the  procedures  used  to 
indicate  when  these  are  necessary  should  be  prescribed. 

The  control  charts  that  should  be  maintained  and  the  rationale  for  their 
interpretation  should  be  specified.  This  will  include  those  to  be  maintained 
by  the  measurement  laboratory  and  those  by  others,  such  as  a reference 
laboratory  for  example.  All  control  charts  should  be  kept  on  as  close  to  a 
real-time  mode  as  possible  so  that  deviations  from  statistical  control  can  be 
quickly  detected. 

The  control  samples  to  be  used  such  as  replicates,  internal  reference 
samples.  Standard  Reference  Materials,  blinds,  and  double  blinds  should  be 
specified.  In  doing  so,  due  consideration  should  be  given  to  the  model  (to 
ensure  its  appropriateness)  and  the  data  base  (to  ensure  its  adequacy). 

15.8  RELEASE  OF  DATA 

This  section  should  specify  the  requirements  that  the  data  must  meet  before 
it  may  be  released  by  the  originating  laboratory  and  the  criteria  that  the 
receiving  organization  will  use  to  accept  data.  The  persons  authorized  to 
judge  the  above  should  be  designated  also.  When  sensitive  data  are 
involved,  the  chain  of  official  release  should  be  defined. 
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Table  3.  Protocols  for  Specific  Purposes* 


What  they  are 

protocols  to  define  what  is  to  be  done  in  a specific  problem/ 
project/ task/program. 

How  prepared 

by  responsible  authority 

may  concern  recurring  problems 

may  be  tailored  to  a specific  problem;  approval  by  management/ 
client  desirable  in  this  case 


Content 

designation  of  principal  investigator 
definition  of  problem 
specification  of  model 
specification  of  sample 
specification  of  data  base 
specification  of  methodology 

references  to  GLPs,  SOPs,  other  PSPs  as  appropriate 
specification  of  any  deviations  or  exceptions  from  above 
specification  of  controls 
control  charts  to  be  maintained 
quality  assessment  procedures 

Release  of  Data 

data  quality  requirements 
quality  assurance  responsibilities 


*Sometimes  called  project  QA  plans  [^5]. 
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16.  QUALITY  ASSURANCE  FOR  A MONITORING  PROGRAM 


This  section  describes  the  quality  assurance  aspects  of  a monitoring  program 
[51].  Monitoring  programs  may  range  in  scope  from  those  involving  a number 
of  participants  over  an  extended  period  of  time  to  individual  laboratories 
seeking  limited  objectives.  In  any  case,  the  program  or  project  needs  to  be 
well  planned  and  the  data  must  be  reliable  if  conclusions  are  to  be  drawn. 

The  following  discussion  is  written  with  large  programs  in  mind  but  applies 
in  principle  to  all  such  programs,  regardless  of  size. 

An  adequate  quality  assurance  program  is  a critical  requirement  for  any 
monitoring  program  and  the  need  increases  with  the  number  of  participants  and 
the  complexity  of  the  data  base.  Experience  has  shown  that  adequate  quality 
assurance  practices  cannot  be  assumed,  even  for  a single  laboratory.  By  the 
laws  of  probability,  the  chance  of  faulty  data  will  increase  rapidly  as  the 
number  of  participants  increases.  Monitoring  programs  are  often  initiated  in 
response  to  what  is  believed  to  be  an  urgent  problem.  The  assumed  urgency 
may  override  prudent  planning  in  the  haste  to  obtain  information.  However, 
if  the  planning  and  quality  assurance  procedures  are  faulty,  the  resulting 
data  can  be  questionable,  misleading,  and  even  worthless. 

The  planning  for  any  project  and  especially  for  a major  monitoring  program 
should  proceed  through  the  steps  outlined  in  the  following  and  illustated  in 
figure  7 (page  30). 

The  problem  which  is  intended  to  be  solved  by  the  monitoring  data  must  be 
carefully  defined.  Precisely  worded  questions  should  be  formulated  that  need 
quantitative  answers.  At  the  same  time,  the  related  questions  which  are  not 
expected  to  be  answered  (that  are  not  addressed)  by  the  measurement  program 
may  need  to  be  stated.  In  other  words,  there  should  be  no  uncertainty,  at 
the  start,  as  to  what  is  hoped  to  be  accomplished.  This  will  not  preclude 
inconclusive  results  but  such,  as  they  occur,  should  clarify  the  directions 
that  should  be  pursued  to  obtain  conclusive  answers.  In  the  case  of  a 
controversy,  all  parties  concerned  should  agree  that  the  questions  are  the 
right  ones.  Until  such  agreement  is  reached,  it  may  be  futile  to  obtain  data 
since  it  will  be  impossible  to  reach  any  agreement  as  to  what  has  been 
accomplished. 

The  model  for  the  monitoring  program  also  must  be  established  in  advance. 

This  defines  the  data  base  as  to  kind  and  quality  and  specifies  the  samples 
that  are  to  be  analyzed.  If  comparisons  are  to  be  made  with  control  sites, 
such  comparisons  should  be  based  on  samples  in  sufficent  number,  analyzed 
preferably  by  the  participants,  to  make  the  results  statistically 
inter pretable.  The  model  should  describe  how  all  data  are  to  be  reduced  and 
evaluated. 

Preliminary  information  ordinarily  will  be  required  to  assist  in  framing 
appropriate  questions  and  developing  an  adequate  model.  If  such  information 
is  incomplete  (and  even  if  not)  the  questions  and  model  will  need  to  be 
re-examined,  on  the  basis  of  initial  data,  to  confirm  their  credibility,  or 
to  indicate  revisions  as  necessary.  In  brief,  the  design  of  this  as  well  as 
other  parts  of  the  program  needs  to  be  an  iterative  process  and  will  need 
periodic  re-examination  to  ensure  its  credibility  and  effectiveness.  The 
worst  thing  that  can  be  done  is  to  hastily  conceive  a measurement  program, 
blindly  carry  it  out,  and  then  try  to  make  something  of  the  data. 
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The  next  logical  step  is  planning.  Based  on  the  model,  plans  should  be  made 
for  all  aspects  of  the  measurement  process,  including;  sampling;  measure- 
ment; calibration;  and  quality  assurance.  The  PSP  should  be  developed  at 
this  stage,  however,  this  too  should  be  an  iterative  process  and  revisions 
may  be  necessary,  based  on  operating  experience. 

The  PSP  should  clearly  define  what  control  charts  are  to  be  maintained  and 
the  requirements  for  demonstration  of  statistical  control  of  the  measurement 
process  at  the  time  that  data  are  obtained  for  submission.  The  PSP  should 
also  address  the  subject  of  qualitative  identification  of  the  measurand  and 
should  state  what  confirmatory  evidence  is  needed  to  support  it. 

When  the  limit  of  detection  is  of  concern,  the  maximum  acceptable  value  for 
the  method  limit  of  detection  (MDL)  (19)  should  be  specified  in  the  PSP  and 
the  frequency  that  the  laboratory  should  confirm  its  MDL  should  be  stated. 

Because  problem  solving  is  essentially  hypothesis  testing,  competent 
statistical  assistance  may  be  needed  in  problem  definition,  modeling,  and 
planning.  It  is  a common  mistake  to  seek  such  advice  after  all  data  have 
been  collected  and  attempts  are  being  made  to  interpret  them. 

Intercalibration  is  an  essential  part  of  any  multilaboratory  monitoring 
program.  That  is  to  say,  formal  activities  should  be  undertaken  to  ensure 
comparability  and  compatibility  of  the  data  produced.  The  first  require- 
ment is  that  the  participating  laboratories  must  have  basic  demonstrated  com- 
petence in  the  specific  measurement  area  (51).  Then,  they  must  maintain  this 
competence  throughout  the  program.  Systems  and  performance  audits  before  and 
during  the  program  are  essential.  The  latter  need  to  be  well  planned  with  a 
rapid  turn-around  of  the  results  so  that  appropriate  corrective  actions  can 
be  taken  as  they  are  required  to  eliminate  measurement  problems.  The  program 
authority  has  the  responsibility  to  conduct  the  intercalibration  program  and 
it  should  provide  technical  assistance,  as  appropriate,  to  correct  signifi- 
cant measurement  problems.  The  use  of  a reference  laboratory  (see  Section 
17)  is  an  excellent  way  to  accomplish  this  purpose. 

An  absolute  requirement  for  all  monitoring  programs  is  peer  review,  prior  to 
their  initiation.  At  any  time  that  major  changes  are  required,  peer  concui — 

rence  should  be  obtained  as  well.  All  aspects  of  the  program the  model, 

the  plans,  the  measurement,  the  PSP's — should  be  scrutinized  and  all  ques- 
tions resolved  before  the  program  is  activated.  In  the  case  of  adversarial 
situations,  all  opposing  factions  should  be  represented  in  the  review 
process . 

A procedure  similar  to  that  known  by  the  legal  profession  as  "stipulation" 
may  be  followed  to  advantage  in  developing  monitoring  programs.  In  this,  the 
attorneys  involved  agree  in  advance  on  how  legal  proceedings  are  to  be  con- 
ducted in  a particular  litigation.  Similarly,  in  monitoring  programs,  all 
parties  concerned  should  have  the  opportunity  to  agree  on  what  is  to  be  done 
also  (what  cannot  be  done  in  some  cases)  and  on  any  limitations  based  on  such 
considerations  as  time,  cost,  availability  of  samples,  and  the  state-of-the- 
art  of  measurement  and  interpretation  of  the  data.  Unless  there  can  be  a 
reasonable  consensus  on  such  matters,  in  advance,  the  outcome  of  the  work, 
even  when  properly  done,  can  be  disappointing  and  raise  questions  ranging 
from  minor  to  major.  In  the  case  of  the  latter,  one  should  ask  whether  th^^ 
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proposed  work  is  worthwhile  and  whether  research  or  effort  should  be  devoted 
to  delineate  the  problem  and  methods  for  its  solution,  before  a monitoring 
program  is  activated. 

A monitoring  program  must  recognize  that  quality  of  data  is  a paramount 
consideration  and  must  devote  sufficient  resources  to  accomplish  this  goal. 
When  purchasing  measurement  services,  realistic  quality  performance  specifi- 
cations and  quality  assurance  practices  should  be  established  and  vendors 
should  be  held  accountable  for  following  them.  Laboratories  should  not  be 
paid  for  data  of  insufficient  or  undocumented  quality  [51]. 
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17.  FUNCTIONS  OF  A REFERENCE  LABORATORY 


Large  monitoring  programs  may  find  it  advantageous  to  use  a reference 
laboratory  (RL)  to  provide  services  which  would  increase  confidence  in  the 
data,  intercalibrate  the  measurement  laboratories  within  the  network, 
maintain  continuous  surveillance  of  the  measurement  program,  and  advise  on 
corrective  actions  as  required.  The  reference  laboratory  may  provide  a gamut 
of  services  ranging  from  advice  to  full  responsibility  for  operation  of  a 
monitoring  program.  The  reference  laboratory  concept  is  presented  in  outline 
form  in  appendix  A. 5 and  is  summarized  in  the  following. 

Several  possible  interrelationships  of  a reference  laboratory  to  others  in  a 
monitoring  program  are  illustrated  in  figure  8.  Obviously,  the  intercalibra- 
tion lines  must  be  strong  if  all  data  are  to  have  peer  status  and  are  to  be 
compatible.  The  strength  of  the  other  bonds  will  depend  on  the  degree  of 
responsibility  delegated  to  the  reference  laboratory.  The  latter  may  have 
full  responsibility  for  the  entire  program,  in  which  case  the  direct  lines 
between  the  user  and  the  measurement  laboratories  could  be  virtually  non 
existent.  When  the  user  assumes  major  management  responsibilities,  the  lines 
between  the  reference  laboratory  and  the  measurement  laboratories  accordingly 
will  be  weaker. 

An  effective  reference  laboratory  should  have  the  following  qualifications. 

It  should  have  recognized  competence  and  a dedication  to  high  quality  work. 
Preferably,  it  should  be  a recognized  center  of  excellence  with  broad 
interdisciplinary  capability  and  experience.  It  should  have  a reputation  for 
impartiality  and  objectivity.  Furthermore,  and  as  a major  requirement,  it 
must  have  an  effective  ongoing  quality  assurance  program  to  serve  as  a model 
for  the  measurement  laboratories,  to  back-up  its  technical  opinions,  and  to 
provide  authoritative  support  for  its  own  data  outputs. 

A well-qualified  reference  laboratory  can  provide  some  or  all  of  the 
following  functions  to  a monitoring  program  (see  table  4).  It  may  provide 
research  and  development  support  with  respect  to  both  methodology  and 
reference  materials.  With  respect  to  the  former,  it  may  validate  proposed 
methodology  for  use  in  the  monitoring  program,  identify  gaps  in  the 
measurement  process,  and  develop  new  or  improve  exisiting  methods  as  needed. 
It  may  conduct  collaborative  tests  to  do  some  of  the  above.  Concerning 
reference  materials,  it  can  investigate  various  options  and  validate  and 
recommend  materials  for  uses  ranging  from  routine  measurement  control  to 
proficiency  testing  of  participating  laboratories.  Again,  when  suitable 
reference  materials  are  not  available,  the  reference  laboratory  may  be  able 
to  develop  ones  appropriate  for  the  specific  monitoring  program. 

The  reference  laboratory  may  serve  in  a referee  laboratory  capacity.  As 
such,  it  can  analyze  (or  re-analyze)  a fraction  of  the  total  samples  to 
verify  measurement  consistency  and  to  corroborate  key  data  to  increase 
confidence  in  it.  The  reference  laboratory  may  engage  in  trouble  shooting, 
as  necessary,  with  respect  to  sampling,  methodology,  or  other  aspects  of  the 
program. 
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Table  Functions  of  a Reference  Laboratory  in  National 
Measurement/Monitoring  Programs 
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Sample  Flow,  Distribution  by  User 

Data  Flow 

QA  Sample  Flow 

Alternate  Sample  Flow,  Distribution  by  Ref.  Lab. 
Intercalibration  Program 


Figure  8.  Cooperative  Measurement  Program 
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The  quality  assurance  function  can  range  from  furnishing  advice  to  oversight 
of  the  entire  activity  (see  figure  9).  The  reference  laboratory  may  be  used 
to  review  and  pre-validate  the  quality  assurance  program  and  to  demonstrate 
that  it  is  credible  and  can  be  expected  to  be  effective.  Likewise  the  PSP 
for  the  entire  monitoring  program  can  be  designed  and/or  validated  by  a 
competent  reference  laboratory.  It  may  conduct  proficiency  tests  of  the 
measurement  laboratories  including  those  to  qualify  potential  participants 
and  to  verify  their  continuing  competence  throughout  the  program.  The 
circulation  of  samples  and  evaluation  of  inter-calibration  test  results  can 
be  a prime  responsibility.  Belatedly  but  possibly  separately,  it  may  provide 
statistical  services. 

The  reference  laboratory  may  be  given  a training  responsibility.  It  is 
recommended  that  larger  monitoring  programs  should  conduct  workshops,  before 
and  at  intervals  during  a monitoring  program,  to  indoctrinate  participants, 
to  maintain  awareness  of  programmatic  goals  and  progress,  and  to  introduce 
any  necessary  changes  in  the  program.  The  reference  laboratory  could  assume 
responsibilities  for  such  activities. 

The  reference  laboratory  and/or  the  user  should  maintain  a continuous  alert 
for  deficiencies  in  the  monitoring  program  that  would  be  correctible  by 
training  and  provide  such  on  an  individual  laboratory  or  group  basis  as 
appropriate.  Such  training  conceivably  could  be  in  the  use  of  the  methodo- 
logy for  measurement  and  sampling  or  in  quality  assurance  aspects  of  the 
monitoring  program. 

In  a continuing  monitoring  program,  the  reference  laboratory  also  could  have 
the  responsiblity  to  groom  a successor  or  to  develop  a line  of  succession,  as 
described  below,  to  provide  for  internal  surveillance  of  the  program. 

Because  of  its  important  function,  the  reference  laboratory  should  be 
selected  with  great  care.  It  may  be  chosen  from  laboratories  not  regularly 
engaged  in  the  monitoring  program  or  it  may  be  one  of  the  measurement 
laboratories.  The  former  approach  is  recommended  in  the  case  of  extensive 
programs  and  the  latter  is  possible  in  small  programs  or  as  a mechanism  for 
maintaining  a larger  program  after  it  is  well  established.  In  such  a case, 
the  reference  laboratory  could  be  given  the  responsibility  to  develop  a 
training  program  that  would  provide  for  internal  surveillance  by  the 
participants  on  a rotational  basis. 

The  initial  selection  of  a reference  laboratory  and  of  any  internal 
successors  is  based  on  finding  a laboratory  with  unquestionable  qualifica- 
tions and  this  must  receive  consensus  approval.  The  reference  laboratory 
must  maintain  a high  degree  of  credibility  throughout  its  tenure  since  it 
will  be  quite  influential  in  developing  and  implementing  the  monitoring 
policy . 
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Reference  Laboratory 


^•igure  9.  Quality  Assurance  Function  of  a Reference  Laboratory 
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APPENDIX  A.  PROGRAM  DOCUMENTATION 


This  appendix  contains  supplemental  material  which  may  be  useful  in  the 
development  of  documents  related  to  the  quality  assurance  program  of  a 
specific  laboratory.  The  material  is  generic  in  content  and  is  believed  to 
be  widely  useful.  However,  each  laboratory  should  consider  its  applicability 
and  will  need  to  develop  the  specific  content  most  useful  to  its  own 
measurement  program. 

The  material  in  this  appendix  consists  largely  of  outlines  that  can  be 
followed  when  preparing  specific  documentation.  It  is  recommended  that  a 
laboratory  or  organization  which  uses  such  should  address  each  item  in  an 
outline,  and  prepare  text  as  appropriate  for  its  own  situation. 

An  exception  to  the  outline  format  is  Appendix  A.1  which  contains  the  text  of 
a program  document  that  one  laboratory  has  found  to  be  appropriate  to  its 
measurement  situtation.  This  document  is  presented  as  an  example  of  such  a 
document  and  is  presented  only  for  guidance. 


A.1  EXAMPLE  QUALITY  ASSURANCE  PROGRAM  DOCUMENT 

1.  Policy 

The  objective  of  the  measurement  program  of  the  

is  to  produce  high-quality  research  and 

analytical  measurements  and  data  which  are  accurate,  reliable,  and  adequate 
for  the  intended  purpose  in  a cost-effective  manner.  To  this  end,  the 
management  is  dedicated  to  the  encouragement  of  excellence  in  measurement  and 
to  provide  the  physical  and  mental  environment  conducive  to  its  achievement. 

2.  Purpose 

The  purpose  of  this  document  is  to  formalize  the  quality  assurance  practices 
that  have  evolved  during  the  past  and  which  are  currently  operative  within 

the  . A further  purpose  is  to  inform  users  of  the  services  of 

the  conditions  and  practices  related  to  the  services  and  data  provided. 

Due  to  the  diversity  of  programs,  it  may  not  be  possible  to  set  forth  all 
inclusive  statements  nor  to  establish  a rigid  quality  assurance  plan.  The 
management  encourages  the  developement  and  use  of  the  best  measurement 
practices  as  dictated  by  each  measurement  situation.  However,  it  is  the 
policy  to  follow  the  procedures  set  forth  herein  to  the  extent  possible  and 
to  document  the  experimental  details  in  each  individual  case  and  maintain 
them  as  a matter  of  record. 

3.  General  Aspects 

3.1  Preliminary  Requirements 

All  analytical  measurements  will  be  done  according  to  a plan  individually 
developed  and  optimized  for  each  situation.  The  requirements  of  each  problem 
will  be  studied  and  thoroughly  understood  beforehand,  in  order  to  assure  that 
the  measurements  planned  will  be  adequate  both  in  kind,  number  and  quality. 
All  measurements  are  made  using  standard  methods,  methods  with  peer 
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recognition,  methods  developed  in  the  specific  laboratory,  or  those  mandated 
by  legal  or  other  requirements.  No  method  will  be  used  for  data  output 
purposes  until  it  is  known  to  be  applicable  and  competence  has  been  acquired 
in  its  use.  If  or  when  it  appears  that  available  techniques  are  not  adequate 
to  solve  a particular  problem,  the  user  of  the  services  will  be  informed  of 
any  preliminary  investigations  or  research  required,  including  the  estimated 
costs . 

Demonstration  and  documentation  of  attainment  of  statistical  control  of  the 
measurement  process  shall  be  a prerequisite  for  reporting  any  data,  whether 
of  a research  or  routine  nature. 

All  laboratory  equipment  shall  be  maintained  in  a reasonable  and  proper  state 
of  repair.  Space  shall  be  maintained  according  to  established  rules  of 
cleanliness,  order,  and  efficiency  to  facilitate  the  measurements  and  to 
protect  the  health  and  safety  of  the  laboratory  personnel.  No  equipment  is 
operated  until  it  is  first  ascertained  that  it  is  in  a safe  and  reliable 
state  and  then  only  by  personnel  who  have  been  thoroughly  trained  and  have 
been  duly  qualified  as  operators. 

Work  done  for  another  organization  will  adhere  to  any  specified  project  plan 
or  quality  assurance  program  to  the  extent  possible.  Whenever  such  direc- 
tives differ  from  on-going  policy,  such  differences  will  be  reconciled  before 
work  is  initiated  or  whenever  they  are  first  recognized. 

3.2  Experimental  Plan 

All  experimental  work  is  to  be  done  according  to  a written  plan  developed  in 
advance  of  the  actual  work.  Where  applicable  and  appropriate,  the  plan  is 
developed  in  consultation  with  the  end-user  of  the  analytical  data  and  the 
statistician  who  will  assist  in  interpreting  the  data  or  reviewing  the  con- 
clusions. The  plan  includes  the  sampling  strategy,  the  sampling  procedure, 
the  calibration  process,  the  measurement  procedure,  and  the  data  handling  and 
validation  process.  All  procedures  used  should  be  documented  and  the 
documentation  should  be  available  for  review  by  users  of  the  data.  Standard 
Operations  Procedures  (SOP's)  are  used  to  the  extent  practical.  When  other 
procedures  are  required,  their  adequacy  is  verified  before  use  in  a 
measurement  process. 

3.3  Reagents 

Reagents  are  defined  as  any  substances  used  to  dissolve,  disperse,  extract, 
react  with,  or  dilute  any  sample  or  analytical  component  of  a sample.  Strict 
control  of  reagents  shall  be  maintained  to  minimize  their  contamination  or 
degradation.  Dilutions  or  solutions  prepared  from  such  reagents  shall  be 
clearly  labeled  and  dated.  The  shelf  life  shall  be  designated  and  strictly 
observed . 

The  highest  quality  analytical  reagents,  consistent  with  the  end  use,  shall 
be  used  in  all  analytical  work.  Records  will  be  kept  of  all  reagents  used  in 
every  critical  analysis  and  each  analytical  plan  shall  include  procedures  to 
carefully  control  the  quantities  used  and  to  minimize  and  correct  for  the 
analytical  blank  to  the  extent  possible. 
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3.4  Laboratory  Safety 


The  policy  of  the  is  as  follows:  General  safety  rules  as 

developed  by  will  be  followed.  Special  safety  rules  in  addtion 

to  the  above  will  be  part  of  any  GMP  for  a specific  measurement  competence. 

3.5  Disposal  of  Toxic  Substances 

The  potential  hazard  of  each  substance  used  regularly  or  occasionally  will  be 
investigated  as  a prerequisite  for  its  use.  No  potentially  hazardous 
material  will  be  used  until  it  has  been  approved  for  use  by  at  least  one 

level  of  supervision  above  the  user.  The  Safety  Office  will  be 

consulted  in  each  case  as  appropriate  and  disposal  will  be  by  that  office  or 
in  accordance  with  its  approved  procedures. 

4.  Samples 

It  is  recognized  that  sampling  is  one  of  the  most  critical  operations  in 
analytical  measurement.  The  analyst  can  only  report  on  the  measurement  made 
on  a specific  sample  at  a specific  time.  Often  this  is  not  the  information 
ultimately  desired  but  rather  the  projection  of  that  information  in  time  and 
space  and  its  relation  to  the  population  from  which  the  sample  was  drawn. 

The  sampling  strategy,  methodology,  subsampling,  chemical  and/or  physical 
treatment,  preservation,  and  storage  are  logically  parts  of  the  analytical 
plan  and  must  be  developed  and  specified  for  each  situation. 

Sampling  plans  and  measurement  procedures  must  be  adequate  to  permit 
evaluation  of  sample  homogeneity  and/or  population  distributions  when  these 
are  important  considerations. 

Any  limitations  on  the  analytical  results  due  to  the  sample  must  be  clearly 

specified  in  the  analytical  report.  It  is  the  policy  of  the  

that  the  integrity  of  samples  should  be  unquestionable.  All  samples  are  to 
be  identified  in  an  unambiguous  manner  and  fully  documented  in  the  respective 
record  books.  Samples  shall  be  stored  to  minimize  contamination  and/or  dete- 
rioration while  in  the  and  shall  be  retained  as  long  as  neces- 

sary in  the  judgement  of  the  supervisor  or  delegated  authority.  Adequate 
records  shall  be  kept  to  establish  a chain  of  custody  and  to  ensure  their 
integrity,  identification,  and  locatability  at  all  times.  When  necessary  or 
mandated,  a legal  chain-of-custody  shall  be  established  including  the 
designation  of  a sample  custodian. 

5.  Analytical  Methodology 
5.1  General 

All  analytical  measurements  shall  be  made  using  methodology  appropriate  for 
the  specific  purpose.  Each  measurement  method  will  be  described  in  writing 
before  use  in  the  generation  of  analytical  data.  This  serves  the  purpose  of 
recording  the  procedure  used  and  also  ensures  that  a sequence  of  measurements 
will  be  made  using  the  same  procedure,  which  is  a prerequisite  for  statisti- 
cal treatment  of  the  resulting  data.  Any  significant  changes  in  a measurement 
procedure  shall  be  documented  and  notebook  reference  shall  make  clear  when 
such  changes  are  made. 


When  changes  are  made  to  an  SOP,  the  specific  changes,  together  with  the 
rationale  for  them,  shall  be  documented  and  clearly  Indicated  whether  the 
change  Is  permanent  or  pertains  to  a specialized  use.  Any  significant  changes 
will  be  considered  as  generating  a new  method  which  will  need  to  be  validated 
before  It  may  be  used  for  definitive  measurements. 

The  written  procedure  will  Include  the  following  Information,  as  applicable: 

a.  Sample  preparation  and  treatment 

b.  Chemical  operations 

c.  Calibration  procedure 

d.  Measurement  procedure 

1.  Instrumental  conditions 

2.  Instrumental  adjustments 

e.  Data  handling  (See  4.3) 

References  to  Standard  Operations  Procedures  (SOP’s)  when  they  are  used  will 
satisfy  this  requirement  for  documentation. 

5.2  Standard  Operations  Procedures  (SOPs) 

Standard  operations  procedures  (SOPs)  designate  those  measurement  procedures 
or  sub-steps  which  have  been  established  by  consensus  actions  of  standards 
organizations  or  those  reduced  to  a standard  format  and  demonstrated  by  con- 
sistent use  in  the  laboratory  to  be  the  preferred  way  to  do  a measurement  or 
operation  of  concern. 

Measurement  procedures  or  subsets  thereof  which  are  used  in  a recurring 
manner  should  be  written  as  SOPs  to  which  reference  can  be  made  in  the  event 
of  future  use. 

SOPs  will  be  written  in  a standard  format  when  possible.  The  format  of 
preference  is  that  used  by  ASTM. 

5.3  Calibration 

5.3.1  Calibration  generally  refers  to  physical  measurements  made  on  objects, 
apparatus,  or  equipment  to  verify  or  provide  corrections  to  the  experimental 
data  it  provides.  Examples  include  masses,  volumetric  apparatus,  and 
flowmeters.  Records  of  such  calibrations,  whether  made  externally  or 
internally,  will  be  maintained  in  a central  location  within  the  corresponding 
division.  Copies  are  also  made  available  to  the  users  of  the  calibrated 
articles  and  supervisors  will  be  responsible  to  see  that  the  latest 
calibration  information  is  applied  in  each  pertinent  situation. 

5.3.2  Standardization  is  a type  of  chemical  calibration  and  refers  to  those 
operations  in  which  instrument  response  (in  its  broadest  interpretation)  is 
related  to  analytical  data.  Examples  include  the  titer  of  a solution,  an 
absorbance-concentration  curve,  and  the  spanning  of  a linear-direct  reading 
chemical  analyzer.  Except  for  the  latter,  such  data  are  ordinarily  recorded 
in  notebooks.  The  reliability  of  many  measurements  is  critically  dependent 
on  the  quality  of  the  chemical  calibrants  used  in  the  standardization 


50 


process.  This  may  involve  obtaining  primary  standards  and  the  preparation  of 
secondary  standards.  In  such  cases,  labeling,  record  of  preparation, 
expiration  date,  and  special  precautions  for  use  are  prime  considerations. 

In  some  cases,  calibrants  may  be  obtained  from  external  sources  such  as  a gas 
manufacturer . 

5. 3. 3 It  is  the  responsibility  of  each  analyst  to  see  that  the 
concentrations  of  all  chemical  standards  used  are  verified  by  reliable  met- 
hods. The  procedures  used  for  such  verification,  as  well  as  the  use  of 
standards  shall  be  fully  documented. 

5.3.^  Calibrations  and  standardizations  shall  be  made  with  sufficient 
frequency  to  minimize  uncertainties  in  the  analytical  results  due  to 
uncertainties  of  calibration  and/or  standardization. 

5.3.5  All  calibrations  and  standardizations  shall  be  fully  documented  and 
the  data  shall  provide  clear  traceability  to  the  calibrations  related 
thereto . 

5. ^  Data  Handling 

Descriptions  of  each  analytical  method  shall  include  procedures  for  handling 
data.  Because  the  procedures  may  be  common  to  many  of  the  measurements  in  a 
given  laboratory,  this  may  be  best  described  as  a GLP.  The  following 
subjects  should  be  addressed,  as  appropriate: 

• Procedures  used  in  the  calculation  of  data;  corrections  required; 

adjustment  to  standard  conditions;  normalization  of  data;  computer 
programs . 

• Statistical  procedures  used  to  report  data;  method  for  evaluating 

limits  of  uncertainty. 

• Corrections  for  sytematic  errors. 

• Source  of  all  constants  used  in  calculations,  such  as  Faraday 

constant,  atomic  weights,  molecular  weights. 

• Checks  on  accuracy  of  calculations;  use  of  significant  figures; 

propagation  of  errors;  analysis  of  variance. 

6.  Laboratory  Records 
6.1  General 

It  is  the  responsibility  of  each  individual  in  the  to  maintain 

complete  records  of  her/his  work  in  the  detail  and  format  prescribed  by  the 
particular  organizational  unit.  The  minimum  requirement  shall  be  that  all 
significant  experimental  work  shall  be  described  in  sufficient  detail  to  make 
it  understandable  to  one  versed  in  the  art,  yet  not  directly  connected  with 
the  work. 

All  laboratory  record  books  are  the  property  of  the  and  may  not  be 

removed  from  the  premises  without  permission  of  higher  authority.  Records 
are  to  be  suitably  protected  and  stored  to  conform  with  the  general  regula- 
tions of  the  . Individuals  who  wish  to  maintain  personal  records 

may  do  so,  but  these  shall  be  in  addition  to  the  complete  and  permanent 
records  mentioned  above. 
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6.2  Measurement  Data 


All  experimental  data  will  be  recorded  in  bound  laboratory  record  books. 
Transcriptions  shall  be  avoided  wherever  possible.  The  data  shall  be  suitably 
annotated  with  reference  to  experimental  details,  conclusions  and/or  future 
directions  as  needed  to  positively  identify  it.  The  data  record  may  be  kept 
in  a general  laboratory  record  book  or  in  a separate  book,  as  approved  by 
supervisors,  but  cross  references  shall  be  made  as  appropriate.  Corrections 
to  recorded  data  shall  be  documented  and  no  erasures  will  be  permitted.  The 
name  of  the  analyst  shall  be  clearly  indicated  so  that  there  is  no  question 
as  to  who  is  responsible  for  the  work. 

Data  in  the  form  of  charts,  instrument  recordings,  and  printouts  will  be 
given  suitable  identification  numbers  and  maintained  in  a manner  consistent 
with  that  indicated  for  written  records.  Laboratory  record  books  will  refer 
to  the  location  and  identification  number  of  such  records. 

In  view  of  the  possible  legal  implications  of  much  of  the  data,  all  data 
shall  be  recorded  in  such  a way  as  to  preclude  its  discreditation  at  any 
time.  Any  data  obtained  for  use  of  patent  applications  or  for  other  legal 
purposes  may  need  to  be  substantiated  by  witnesses.  The  rules  promulgated  by 
the  legal  advisor  will  be  followed  in  such  cases. 

6.3  Maintenance  Records 

Records  of  maintenance  of  equipment  shall  be  suitably  recorded.  Routine 
maintenance  may  be  indicated  by  labels  on  equipment.  Maintenance  which 
results  in  modification  of  equipment  must  be  described  in  sufficient  detail 
and  recorded  in  the  operational  manual  for  the  particular  equipment  (see 
6.4). 

6.4  Instrument  Manuals 

A copy  of  the  operational  manual  supplied  with  equipment  shall  be  maintained 

in  the  or  other  locations  as  approved.  In  the  latter  case, 

will  maintain  a record  of  where  the  manuals  are  located.  In 

addition,  requires  that  a copy  be  placed  in  the  Library.  The  file 

will  contain  descriptions  of  any  modifications  that  will  alter  either  the 
operation  or  servicing  instructions  furnished  by  the  supplier.  As  appro- 
priate, the  file  will  contain  additional  information  on  operational 
procedures  and/or  servicing  as  may  be  generated  as  a result  of  laboratory 
experience  with  the  equipment. 

Records  similar  to  those  described  above  will  be  prepared  and  maintained  for 
equipment  built  within  the  laboratory  or  within  . 

7.  Control  Charts 

Control  charts  are  recognized  as  a prime  means  for  documenting  the 
statistical  control  of  a measurement  process. 

It  is  the  policy  of  the  that  each  measurement  competence  shall 

develop  and  maintain  appropriate  control  charts  for  sub-operations  and/or 
overall  operations,  where  feasible. 


52 


It  is  the  responsibility  of  the  senior  scientist  in  each  competence  area  to 
identify  the  kinds  of  control  charts  that  will  be  most  useful.  When  control 
charts  are  specified  in  a given  protocol,  they  shall  be  maintained  in  a 
real-time  mode  as  far  as  possible  and  shall  be  the  basis  for  corrective 
actions  when  indicated. 

8.  Quality  Assessment 

8.1  Data  Verification 

All  data  reported  must  include  a statement  of  the  uncertainty  pertaining  to 
it,  and  the  work  plan  must  include  a means  for  the  determination  of  such 
limits.  In  addition,  all  data  will  be  verified  by  one  or  more  of  the 
following  methods: 

1.  Simultaneous  measurement  of  an  SRM  or  other  certified  reference 

sample 

2.  Measurement  by  an  independent  technique 

3.  Measurement  by  an  independent  operator/apparatus 

4.  Spiking/recovery  technique 

5.  Replicate  measurements  by  a recognized  technique. 

The  verification  data  or  a summary  thereof  will  become  a part  of  the  test 
report . 

The  signer  of  each  report,  in  doing  so,  verifies  that  the  quality  assurance 
requirements  associated  with  it  have  been  fully  met. 

8.2  Any  significant  uncertainties  in  the  measurement  process  or  in  the  data 
produced  by  it  will  be  resolved  before  data  may  be  reported  or  released. 

9.  Data  Review/Reporting 

All  data  will  be  subject  to  peer  review  before  external  release.  All 
releases  will  be  in  writing.  Oral,  including  telephone,  releases  are  per- 
mitted when  appropriate,  provided  they  are  subsequently  (within  24  hours) 
confirmed  in  writing,  and  provided  the  data  have  been  previously  reviewed  and 
approved  as  herein  set  forth.  The  lowest  level  of  review  for  release  of  data 
is  by  the  supervisor  or  his  designate.  In  the  case  of  sensitive  data,  a 
higher  level  of  review  may  be  required.  Publication  of  data  is  predicated 
upon  editorial  review.  Data  obtained  for  a sponsor  is  considered  to  be  the 
property  of  the  sponsor  and  its  dissemination  is  subject  to  the  sponsor's 
release  (within  the  Freedom  of  Information  Act  principles). 

Reports  will  be  prepared  in  a format  suited  to  the  particular  end  use.  Each 
report  will  contain,  as  far  as  practical,  the  following  information: 

a.  Summary  of  the  problem  investigated 

b.  Complete  description  of  samples 

c.  Description  of  methodology  (or  reference) 

d.  Summary  of  data  and  reference  to  original  data  set(s) 

e.  Interpretation/conclusions 

f.  Attestation 

Signatures  of  analysts(s).  Supervisor  (or  designate(s) ) . 
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10.  Implementation 

Each  and  every  member  of  the  staff  has  responsibility  for  the  implementation 
of  the  quality  assurance  program,  commensurate  with  her/his  technical  and/or 
supervisory  and  administrative  responsibilities.  Specific  responsibilities 
are  described  throughout  this  document.  The  general  aspects  are  summarized 
below. 

Individuals;  It  is  the  responsibility  of  each  individual  to  support  the 
quality  assurance  program  in  principle  and  detail.  Each  member  of  the 
staff  is  responsible  for  the  technical  quality  of  her/his  work.  All 
work  must  be  done  with  the  highest  level  of  integrity  and  professional 
competence.  Records  must  be  kept  meticulously.  Each  member  of  the  staff 
must  be  ever  alert  to  problems  or  sources  of  error  that  could  compromise 
the  quality  of  technical  work. 

Management  Chain;  The  management  has  the  responsibility  for  supervising 
and  administrating  the  quality  assurance  program  and  is  responsible  for 
the  overall  administration  of  the  program  with  responsibilities  detailed 
down  through  the  supervisory  chain.  Specific  responsibilities  are 
critical  elements  of  the  performance  agreements  of  each  supervisory  and 
administrative  official. 

Quality  Assurance  Coordinator;  The  has 

established  the  position  of  Quality  Assurance  Coordinator  for  oversight 
of  its  quality  assurance  program.  The  Coordinator  reports  to  the 
Director  and  has  the  following  responsibilities; 

On-going  review  of  the  individual  quality  assurance  procedures 
within  the  various  units;  provides  assistance  in  the  development 
and  implementation  of  specific  quality  assurance  plans  for  special 
analytical  programs  such  as  work  done  for  interagency  agreements; 
review  of  interim  and  final  reports  with  respect  to  quality 
assurance  aspects  of  analytical  data. 

11.  Research  Investigations 

All  research  investigations  shall  be  carried  out  efficiently,  effectively, 
and  with  a high  degree  of  technical  excellence.  Accurate  data  are  essential 
for  reliable  research  conclusions.  The  requirements  for  acquisition  of  such 
data  in  research  situations  are  similar  to  those  already  described.  Accord- 
ingly, all  research  investigations  shall  be  carried  out  consistent  with  the 
general  principles  described  above.  Quality  control  during  such  investiga- 
tions is  primarily  the  responsiblity  of  the  individual  researchers  and  their 
supervisors,  according  to  established  management  chains  of  supervision  and 
oversight . 

Research  outputs  are  in  the  form  of  reports  and  publications.  The  quality 
assurance  for  such  outputs  is  by  means  of  the  editorial  review  process. 

The  respective  responsibilities  in  this  process  are  as  follows; 
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Investigator/Author ; Accuracy  of  data;  adequate  support  for  all 
conclusions;  full  disclosure  and  description  of  all  significant 
technical  details;  maintenance  of  records  of  all  experimental  work, 
including  data;  safeguarding  all  data  and  records  for  reference  in  the 
case  of  future  questions. 

Middle  Manager;  Technical  review  of  all  work  submitted  as  reports  or 
publications  for  scientific/technical  adequacy  and  reliability; 
certification  to  higher  authority  by  signing  of  test  reports  or 
publication  records  that  the  technical  aspects  are  sound  and  that 
supporting  data  and  records  are  maintained  according  to  prescribed 
policy. 

Upper  Manager;  Editorial  review  of  all  manuscripts,  directly  or  by 
delegation,  to  ascertain  that  their  quality  meets  minimum  standards, 
that  all  reasonable  care  has  been  taken  to  insure  technical  validity, 
and  that  the  contents  may  be  released  for  external  dissemination. 


Approved; 
Date : 


A. 2 DEVELOPMENT  OF  A QA  PROTOCOL  FOR  A SPECIFIC  PURPOSE 

Outline  of  the  Development/Implementation  Process 

Study/Review  Analytical  Problem 
Develop/Review  Model 
Develop/Review  Experimental  Plan 

Sampling 

Measurement /Cali brat  ion 
Data  handling 

Develop/Choose/Review  Appropriate  Technology 
Sampling/Calibration/Measurement 
Test  Proposed  Technology  for  Appropriateness 
Validation  of  methodology 
Identify  Discrete  Steps  in  Technology 

In-depth  Analysis  of  Technology/Methodology  for  Sources  of  Error 
Bias 

Variance 

Develop/Review  Procedures  to  Reduce  Uncertainties  to  Acceptable  Limits 
Develop/Specify/Review  Quality  Control  Procedures 
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GLPs 

GMPs 

SOPs 


Develop/Specify/Review  Quality  Assessment  Techniques 

Reference  Materials  (RMs) 

Control  Samples  (CSs) 

IRMs 

Duplicates/Replicates 

Spikes 

Surrogates 

Frequency  of  Use  of  CSs,  SRMs 
Control  Charts 

Develop/Specify/Review  Overall  QA  plan 

Develop/Specify/Review  Criteria  for 

Qualification/Selection  of  Contract 
Laboratories 

Monitor  Performance 

Reference  Laboratory  Function 

Trouble  Shoot  Follow-up  R & D as  Required 


A. 3 OUTLINE-CONTENT  OF  A QUALITY  ASSURANCE  PROTOCOL  FOR  A SPECIFIC 
PURPOSE  (PSP) 

1 . Introduction 

2.0  Project  Description 

2.1  Statement  of  problem 

Brief  statement  of  problem  with  reference  to  contract,  proposal,  or 
other  documents  where  further  information  can  be  found. 

2.2  Model 

Brief  discussion  of  rationale,  approach,  assumptions 

2.3  Major  Elements 

The  basic  activities  including  sampling,  measurement,  data 
processing  and  any  other  items  that  will  be  pursued  concurrently. 

2.4  Organization 

2.4.1  Sponsoring  organization 

2.4.2  Principal  investigator 
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2.H.3  Organizational  chart  and/or  other  documentation  that  will 
describe  the  respective  responsibilities  of  project/ 
organizational  personnel  with  respect  to  present  project. 

2.4.4  Priorities/Scheduling 

A statement  of  any  special  priorities  and/or  scheduling 
pertinent  to  the  present  project. 

3.0  Work  Plan 

3.1  Sampling 

Kinds  and  numbers  of  samples;  sampling  plan;  sampling  schedule; 
sampling  methodology;  sample  storage,  preservation,  processing; 
chain  of  custody.  Refer  to  section  7.1.1  as  appropriate. 

3.2  Calibration 

Kinds;  frequency;  calibrants  to  be  used  refer  to  section  7.1.2  as 
appropriate. 

3.3  Measurement 

Kinds  of  measurements;  measurement  plan;  measurement  schedule; 
measurement  methodology;  refer  to  section  7.1.2  as  appropriate. 

4.0  Quality  Assurance  Objectives 

4.1  Expected  precision  and  accuracy 

4.2  Implementation  Summary 

A brief  summary  of  the  various  aspects  of  the  quality  assurance 
plan  and  how  it  is  to  be  carried  out.  Designation  of  quality  assu- 
rance responsibilities.  Designation  of  a Quality  Assurance  Officer 
as  pertinent  and  responsibilities. 

5.0  Quality  Assurance  Procedures 

5.1  Preliminary  Exercises 

5.1.1  Statement  of  any  specific  training  required  and  how  it  will 
be  given  and  evaluated. 

5.1.2  Workshop  or  other  indoctrination. 

5.1.3  Performance  Checks 

Statement  of  the  measurements/audits  that  will  be  required 
to  qualify  a laboratory  and  its  personnel  and  verify  their 
capability  to  perform  or  to  meet  minimum  qualification 
requirements . 


57 


5.2  Quality  Assessment  Samples 


5.2.1  Description 

Description  of  QA  samples  that  will  be  used  with  associated 
documentation. 

5.2.2  Use 

5 . 2 . 2 . 1 Mode 

5. 2. 2.2  Frequency 

5.3  Control  Charts 

5.3.1  Specification 

Specification  of  all  charts  to  be  maintained  and  the 
respective  responsibilities  for  them. 

5.3.2  Strategy 

How  control  chart  information  is  to  be  used  to  provide 
quality  assurance;  remedial  actions  to  be  taken  when  neces- 
sary. 

5.4  Data  Validation 

A statement  of  the  requirements  that  must  be  met  before  data  is 
considered  to  be  valid  and  may  be  released  for  internal  or  external 
use,  together  with  who  is  authorized  to  make  such  decisions. 

6.0  Management  Practices 

6.1  Chain-of-Custody 

Procedures  to  be  followed  to  safeguard  samples;  retention 
procedures;  positive  identification  requirements. 

6.2  Document  Control 

Records  required;  numbering/identification  scheme;  distribution; 
document  control;  storage  of  records  and  data. 

6.3  Data  Management 

Responsibilities  for  acquisition;  review  procedures;  release 
procedures . 

6.4  Reporting 

Scope  of  reports;  schedule  for  reporting;  responsibilities  for 
reporting;  responsibilities  for  release. 
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7.0  Appendices 


7.1  Methodology 

7.1.1  Sampling 

Specific  methodology  to  be  used  by  reference;  specific 
modifications  of  existing  methods;  complete  details  for 
unreferenced  methods.  This  shall  include;  equipment  for 
sampling;  method  of  sampling;  pretreatments  such  as  sieving 
or  blending;  drying;  preparation  of  analytical  samples  if 
not  included  in  7.1.2.  (see  3.1). 

7.1.2  Measurement 

Include  the  same  type  of  information  as  described  in  7.1.1.; 
describe  calibrations,  calibrants  to  be  used,  (see  3.2.; 
3.3). 

7.2  Maintenance  of  Facilities  and  Equipment 

7.2.1  Maintenance  Schedules 

7.2.2  Calibration  Schedules 

7.2.3  Trouble  Shooting 

This  shall  include  general  directions  or  diagnosis  of 
trouble  and  corrective  actions  to  be  followed. 


A. 4 OUTLINE  - ORGANIZING  AND  IMPLEMENTATION  OF  A MONITORING  PROGRAM 

Contract/Participant  Laboratory  Requirements 

Pre-Contract /Participation  Requirements* 

In-house  QA  Program 

Developed,  in-place,  and  implemented 
Availability  of  appropriate  facilities  and  equipment** 
Availability  of  trained  personnel 
Prior  experience  with  methodology** 

Prior  experience  with  similar  samples*** 

Pr e-ini ti at ion-of-work  Requirements* 

Pre-inspection/review  of  facilities  and  plans 


* General  specification  should  be  included  in  RFP 

**  Specific  specifications  and/or  options  included  in  RFP 

***  Contractor  should  submit  evidence  of  such  with  his  proposal.  This  can  be 
based  on  laboratory  history  or  on  tests  done  especially  for  this  purpose. 
Initiator  should  have  samples  available  for  distribution  free  or  at  cost 
for  use  of  prospective  contractor/participants. 
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Development  of  specific  QA  plan 
Demonstration  of  competence  on  typical  samples 

Monitoring  Contractor/Participant  Laboratory  Performance* 

Adherence  to  Project  QA  Plan 
Performance  on  replicates/split  samples 
Performance  on  prof iciency-testing  samples 
Maintenance  of  facilities  equipment 
Retention  of  personnel  or  suitable  replacement 

Adherence  to  all  aspects  of  contractual/programmatic  requirements 
Consistency  of  results  with  those  of  referee  laboratory**** 
Consistency  with  other  labs  in  monitoring  groups 

Retention  of  Status 

Retention  of  qualification  status  and  payment  for  services  (as 
pertinent)  contingent  on  providing  such  of  required  quality 


A. 5 OUTLINE-FUNCTIONS  OF  A REFERENCE  LABORATORY  (RL) 


The  following  outline  is  furnished  to  provide  guidance  when  the  need  for  a 
reference  laboratory  (RL)  in  a monitoring  program  is  considered.  The 
requirements  that  an  RL  must  satisfy  are  largely  independent  of  the  functions 
it  may  serve.  The  latter  will  depend  on  the  complexity  of  the  program  and 
extent  to  which  the  various  functions  outlined  can  be  furnished  adequately  by 
others  in  the  program. 

Purpose  of  an  RL 

• To  provide  confidence  in  data 

• To  provide  continuing  surveillance  of  measurement  program 
and  to  advise  on  corrective  actions  as  required 

Functions  of  an  RL 

• R and  D services 

-Validate  existing  methodology 
-Identify  gaps  in  methodologies 
-Develop  or  improve  methodology 

• Reference  Materials 

-Validate  existing  RMs 
-Develop  new  RMs 

f Referee  Laboratory  Services 

-Analyze  percentage  of  samples 
-Corroborate  key  data 
-Trouble  shoot  problems 


****Referee  laboratory  should  have  clear  and  indisputable  peer 

qualifications.  It  should  do  sufficient  work  to  be  able  to  clearly 
defend  all  judgments,  including  both  acceptability  and  inacceptability  of 
performance/ data . 
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• Quality  Assurance  Services 

-Advise/design  QA  program 
-Validate  QA  program 
-Design/Test  PSP  (Program  Plans) 

-Conduct  proficiency  tests 
-Qualify  participants 
-Provide  statistical  services 

• Training 

-Wor  kshops 
-Specific  training 
-Methodology 
-QA  program 
-Succession  training 

Requirements  for  an  RL 

• Recognized  competence 

-Dedication  to  excellence 
-Center  of  excellence 

• Broad  interdisciplinary  capability  and  experience 

• Recognized  impartiality 

• On-going  QA  program  to  back-up  its 

technical  findings 

Selection  of  an  RL 

• Independent  laboratory  having  required  qualifications 

• Group  member 

-Rational  role 

-Must  meet  minimum  requirements 
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APPENDIX  B.  DATA  QUALITY  OBJECTIVES 

The  setting'of  realistic  data  quality  requirements  is  a pre-requisite  for 
planning  a measurement  program.  Data  of  insufficient  quality  will  have 
little  value  for  problem  solving  while  data  of  excess  quality  may  provide 
little  if  any  additional  advantages,  and  may  be  counter-productive  if  extra 
cost  is  involved.  Accordingly,  data  quality  objectives  must  be  established 
based  on  cost-effective  considerations  of  needs  of  the  problem  and  capability 
of  the  measurement  process.  In  addition,  knowledge  of  the  expected  variabil- 
ity of  the  samples  is  needed  to  estimate  the  number  of  samples  to  be  taken 
and  the  number  of  measurements  to  be  made  of  each  sample.  While  the  details 
of  the  planning  process  are  beyond  the  scope  of  the  present  paper,  the 
considerations  involved  are  presented  for  general  information  in  the 
following. 

Qualitative  data  are  commonly  considered  to  require  an  accuracy  of  about  ±30 
percent;  quantitative  data  used  for  hypothesis  testing  should  have  a minimum 
accuracy  requirement  of  about  ±10  percent;  semi -quantitative  data  will  lie 
between  these  limits.  In  questions  of  distinguishing  differences,  a numeri- 
cal factor  of  3 will  be  found  to  be  useful.  For  example,  a Standard  Refer- 
ence Material  certified  to  ±\%  is  expected  to  be  useful  for  identifying 
biases  of  at  least  ±3?.  Multiple  independent  measurements  will  reduce  the 
latter  value  by  a factor  of  l//n.  The  same  principle  may  be  applied  to  the 
identification  of  small  differences  in  other  measurement  situations.  In  gen- 
eral, the  minimum  detectable  difference  of  two  measured  values  (with  95$  con- 
fidence) is  3 U//n,  where  U represents  the  uncertainty  of  the  measured 
results  and  n is  the  number  of  measurements. 

In  the  previous  discussion,  the  uncertainty,  U,  represents  that  due  to  both 
the  measurement  process  (Ujjj)  and  the  sample  variability  (Ug).  Ordinarily, 

U = (u|  + u2)l/2  ^iii  apply.  A detailed  discussion  of  the  propagation  of  the 
uncertainties  due  to  sampling  and  measurement,  together  with  the  cost  con- 
siderations in  designing  sampling  and  measurement  plans  is  given  by  Provost 
(35A). 

The  uncertainty  of  the  sample  composition,  Ug,  is  largely  a statistical 
problem,  although  individual  samples  may  be  biased,  as  well.  The  latter  can 
result  from  contamination,  sample  deterioration,  and  discriminatory  sampling 
procedures.  A well-designed  sampling  plan  should  minimize  all  of  the  above 
sources  of  bias. 

The  uncertainty  of  the  measurement  process  contributes  both  systematic  (B) 
and  random  (R)  uncertainties.  The  former  results  from  such  things  as  impro- 
per calibration,  blanks,  and  non-selectivity  of  the  methodology.  The  latter 
includes  variability  in  control  of  critical  parameters.  All  biases  should  be 
corrected  to  the  extent  possible  so  that  only  uncertainties  in  the 
corrections  remain. 

The  total  uncertainty,  U^,  in  either  the  sample  composition  or  in  its 
measurement  may  be  estimated  as  follows: 

U^  = ± (Ug  + Ug) 
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The  subscripts  indicate  the  several  sources  of  uncertainty  in  the  bias  or 
random  error  as  they  are  individually  identifiable.  All  uncertainties  are 
ordinarily  estimated  at  the  95$  confidence  level.  In  other  than  simple  or 
well-defined  measurement  problems,  the  collaboration  of  a competent 
statistician  is  recommended  to  assist  in  the  development  of  realistic  data 
quality  objectives  and  in  the  design  of  adequate  measurement  and  sampling 
plans . 
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APPENDIX  C.  TERMINOLOGY  USED  IN  QUALITY  ASSURANCE»f 


John  K.  Taylor 

National  Bureau  of  Standards 

Absolute  Method  - A method  in  which  characterization  is  based  entirely  on 
physically  (absolute)  defined  standards. 

Accreditiation  - A formal  process  by  which  a labortory  is  evaluated,  with 
respect  to  established  criteria,  for  its  competence  to  perform  a 
specified  kind(s)  of  measurement.  Also,  the  decision  based  upon  such 
a process.  When  a certificate  is  issued,  the  process  is  often  called 
certification . 

Accuracy  - The  degree  of  agreement  of  a measured  value  with  the  true  or 
expected  value  of  the  quantity  of  concern. 

Aliquant  - A divisor  that  does  not  divide  a sample  into  a number  of  equal 
parts  without  leaving  a remainder;  a sample  resulting  from  such  a 
divisor . 

Aliquot  - A divisor  that  divides  a sample  into  a number  of  equal  parts, 
leaving  no  remainder;  a sample  resulting  from  such  a divisor. 

Analyte  - The  specific  component  measured  in  a chemical  analysis;  also  called 
analate . 

Assignable  cause  - A cause  believed  to  be  responsible  for  an  identifiable 
change  of  precision  or  accuracy  of  a measurement  process. 

Blank  - The  measured  value  obtained  when  a specified  component  of  a sample  is 
not  present  during  the  measurement.  In  such  a case,  the  measured 
value/signal  for  the  component  is  believed  to  be  due  to  artifacts, 
hence  should  be  deducted  from  a measured  value  to  give  a net  value 
due  to  the  component  contained  in  a sample.  The  blank  measurement 
must  be  made  so  that  the  correction  process  is  valid. 

Blind  Sample  - A sample  submitted  for  analysis  whose  composition  is  known  to 
the  submitter  but  unknown  to  the  analyst.  A blind  sample  thus  is  one 
way  to  test  proficiency  of  a measurement  process. 

Bias  - A systematic  error  inherent  in  a method  or  caused  by  some  artifact  or 
idiosyncrasy  of  the  measurement  system.  Temperature  effects  and 
extraction  inefficiencies  are  examples  of  this  first  kind.  Blanks, 
contamination,  mechanical  losses  and  calibration  errors  are  examples 
of  the  latter  kinds.  Bias  may  be  both  positive  and  negative  and 
several  kinds  can  exist  concurrently,  so  that  net  bias  is  all  that 
can  be  evaluated,  except  under  special  conditions. 


*For  other  definitions,  and  particularly  those  related  to  industrial  quality 
assurance,  see;  "Quality  Systems  Terminology",  ANSI/ASQC  Standard  A3"1978, 
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Bulk  sampling  - Sampling  of  a material  that  does  not  consist  of  discrete, 
identifiable,  constant  units,  but  rather  of  arbitrary,  irregular 
units . 

Calibrant  - A substance  used  to  calibrate  or  to  establish  the  analytical 
response  of  a measurement  system. 

Calibration  - Comparison  of  a measurement  standard  or  instrument  with  another 
standard  or  instrument  to  report  or  eliminate  by  adjustment  any 
variation  (deviation)  in  the  accuracy  of  the  item  being  compared. 

Cause-Effect  Diagram  - A graphical  representation  of  the  causes  that  can 

produce  a specified  kind  of  error  in  measurement.  A popular  one  is 
the  so-called  fish  bone  diagram,  first  described  by  Ishikawa,  given 
this  name  because  of  its  suggestive  shape. 

Certification  - See  accreditation. 

Central  Line  - The  long-term  expected  value  of  a variable  displayed  on  a 
control  chart . 

Certified  Reference  Material  (CRM)  - A reference  material  one  or  more  of 

whose  property  values  are  certified  by  a technically  valid  procedure, 
accompanied  by  or  traceable  to  a certificate  or  other  documentation 
which  is  issued  by  a certifying  body. 

Certified  Value  - The  value  that  appears  in  a certificate  as  the  best 
estimate  of  the  value  for  a property  of  a reference  material. 

Chance  Cause  - A cause  for  variability  of  a measurement  process  that  occurs 

unpredictably , for  unknown  reasons,  and  believed  to  happen  by  chance, 
alone . 

Check  Standard  - In  physical  calibration,  an  artifact  measured  periodically, 
the  results  of  which  typically  are  plotted  on  a control  chart  to 
evaluate  the  measurement  process. 

Coefficient  of  Variation  - The  standard  deviation  divided  by  the  value  of  the 
parameter  measured. 

Common  Cause  - A cause  of  variability  of  a measurement  process,  inherent  in 
and  common  to  the  process  itself,  as  contrasted  to  a special  cause 
(defined) . 

Comparative  Method  - A method  which  is  based  on  the  intercompar ison  of  the 
sample  with  a chemical  standard. 

Confidence  Interval  - That  range  of  values,  calculated  from  an  estimate  of 

the  mean  and  the  standard  deviation,  which  is  expected  to  include  *:n*' 
population  mean  with  a stated  level  of  confidence.  Confidence  inter- 
vals in  the  same  context  also  may  be  calculated  for  standard 
deviations,  lines,  slopes,  points. 
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Control  Limit  - The  limits  shown  on  a control  chart  beyond  which  it  is  highly 
improbable  that  a point  could  lie  while  the  system  remains  in  a state 
of  statistical  control. 

Control  Chart  - A graphical  plot  of  test  results  with  respect  to  time  or 

sequence  of  measurement  together  with  limits  within  which  they  are 
expected  to  lie  when  the  system  is  in  a state  of  statistical  control. 

Control  Sample  - A material  of  known  composition  that  is  analyzed 

concurrently  with  test  samples  to  evaluate  a measurement  process 
(see  also  Check  Standard). 

Composite  Sample  - A sample  composed  of  two  or  more  increments  selected  to 
represent  a population  of  interest. 

Cross  Sensitivity  - A quantitative  measure  of  the  response  obtained  for  an 
undesired  constituent  (interferrent ) as  compared  to  that  for  a 
constituent  of  interest. 

Detection  Limit  - The  smallest  concentration/amount  of  some  component  of 

interest  that  can  be  measured  by  a single  measurement  with  a stated 
level  of  confidence. 

Double  Blind  - A sample,  known  by  the  submitter  but  submitted  to  an  analyst 
in  such  a way  that  neither  its  composition  nor  its  identification  as 
a check  sample  are  known  to  the  latter . 

Duplicate  Measurement  ~ A second  measurement  made  on  the  same  (or  identical) 
sample  of  material  to  assist  in  the  evaluation  of  measurement 
variance . 

Duplicate  Sample  - A second  sample  randomly  selected  from  a population  of 
interest  (see  also  split  sample)  to  assist  in  the  evaluation  of 
sample  variance. 

Education  - Disciplining  the  mind  through  instruction  or  study.  Education  is 
general  and  prepares  the  mind  to  react  to  a variety  of  situations. 

Error  - Difference  betwen  the  true  or  expected  value  and  the  measured  value 
of  a quantity  or  parameter. 

Figure  of  Merit  - A performance  characteristic  of  a method  believed  to  be 
useful  when  deciding  its  applicability  for  a specific  measurement 
situation.  Typical  figures  of  merit  include:  selectivity; 

sensitivity;  detection  limit;  precision;  bias. 

Good  Laboratory  Practice  (GLP)  - An  acceptable  way  to  perform  some  basic 

operation  or  activity  in  a laboratory,  that  is  known  or  believed  to 
influence  the  quality  of  its  outputs.  GLP's  ordinarily  are 
essentially  independent  of  the  measurement  techniques  used. 

Good  Measurement  Practice  (GMP)  - An  acceptable  way  to  perform  some  operation 
associated  with  a specific  measurement  technique,  and  which  is  known 
or  believed  to  Influence  the  quality  of  the  measurement. 
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Gross  Sample  (also  called  bulk  sample,  lot  sample)  - One  or  more  increments 
of  material  taken  from  a larger  quantity  (lot)  of  material  for  assay 
or  record  purposes. 

Homogeneity  - The  degree  to  which  a property  or  substance  is  randomly 

distributed  throughout  a material.  Homogeneity  depends  on  the  size 
of  the  subsaraple  under  consideration.  Thus  a mixture  of  two  minerals 
may  be  inhomogeneous  at  the  molecular  or  atomic  level,  but 
homogeneous  at  the  particulate  level. 

Incrememt  - An  individual  portion  of  material  collected  by  a single  operation 
of  a sampling  device,  from  parts  of  a lot  separated  in  time  or  space. 
Increments  may  be  either  tested  individually  or  combined  (composited) 
and  tested  as  a unit. 

Individuals  - Conceivable  constituent  parts  of  a population. 

Informational  Value  - Value  of  a property,  not  certified  but  provided  because 
it  is  believed  to  be  reliable  and  to  provide  information  important  to 
the  certified  material. 

Intercalibration  - The  process,  procedures,  and  activities  used  to  ensure 
that  the  several  laboratories  engaged  in  a monitoring  program  can 
produce  compatible  data.  When  compatible  data  outputs  are  achieved 
and  this  situation  is  maintained,  the  laboratories  can  be  said  to  be 
intercalibrated . 

Laboratory  Sample  - A sample,  intended  for  testing  or  analysis,  prepared  from 
a gross  sample  or  otherwise  obtained.  The  laboratory  sample  must 
retain  the  composition  of  the  gross  sample.  Often  reduction  in 
particle  size  is  necessary  in  the  course  of  reducing  the  quantity. 

Limiting  Mean  - The  value  approached  by  the  average  as  the  number  of 
measurements,  made  by  a stable  measurement  process,  increases 
indefinitely. 

Limit  of  Linearity  (LOL)  - The  upper  limit  of  concentration  or  amount  of 

substance  for  which  incremental  additions  produce  constant  increments 
of  response. 

Limit  of  Quantitation  (LOQ)  - The  lower  limit  of  concentration  or  amount  of 
substance  that  must  be  present  before  a method  is  considered  to  pro- 
vide quantitative  results.  By  convention  LOQ  = lOs^,  where  s^  is  the 
estimate  of  the  standard  deviation  at  the  lowest  level  of 
measurement . 

Lot  - A quantity  of  bulk  material  of  similar  composition  whose  properties  are 
under  study. 

Method  - An  assemblage  of  measurement  techniques  and  the  order  in  which  they 
are  used. 

Outlier  - A value  which  appears  to  deviate  markedly  from  that  for  other 
members  of  the  sample  in  which  it  occurs. 
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Pareto  Analysis  - A statistical  approach  to  ranking  assignable  causes 
according  to  the  frequency  of  occurrence. 

Performance  Audit  - A process  to  evaluate  the  proficiency  of  an  analyst/ 
laboratory  by  evaluation  of  the  results  obtained  on  known  teat 
materials. 

Population  - A generic  term  denoting  any  finite  or  infinite  collection  of 
individual  things,  objects,  or  events;  in  the  broadest  concept,  an 
aggregate  determined  by  some  property  that  distinguishes  things  that 
do  and  do  not  belong. 

Precision  - The  degree  of  mutual  agreement  characteristic  of  independent 
measurements  as  the  result  of  repeated  application  of  the  process 
under  specified  conditions.  It  is  concerned  with  the  closeness 
together  of  results. 

Primary  Standard  - A substance  or  artifact,  the  value  of  which  can  be 

accepted  (within  specific  limits)  without  question  when  used  to 
establish  the  value  of  the  same  or  related  property  of  another  mate- 
rial. Note  that  the  primary  standard  for  one  user  may  have  been  a 
secondary  standard  of  another. 

Probability  - The  likelihood  of  the  occurrence  of  any  particular  form  of  an 

event,  estimated  as  the  ratio  of  the  number  of  ways  or  times  that  the 
event  may  occur  in  that  form  to  the  total  number  of  ways  that  it 
could  occur  in  any  form. 

Procedure  - A set  of  systematic  instructions  for  using  a method  of 

measurement  or  of  sampling  or  of  the  steps  or  operations  associated 
with  such. 

Protocol  - A procedure  specified  to  be  used  when  performing  a measurement  or 
related  operation,  as  a condition  to  obtain  results  that  could  be 
acceptable  to  the  specifier. 

Protocol  for  a Specific  Purpose  (PSP)  - Detailed  instructions  for  the 
performance  of  all  aspects  of  a measurement  program.  This  is 
sometimes  called  a project  QA  plan. 

Quality  - An  estimation  of  acceptability  or  suitability  for  a given  purpose 
of  an  object,  item,  tangible,  or  intangible  thing. 

Quality  Assessment  - The  overall  system  of  activities  whose  purpose  is  to 

provide  assurance  that  the  quality  control  activities  are  being  done 
effectively.  It  involves  a continuing  evaluation  of  performance  of 
the  production  system  and  the  quality  of  the  products  produced. 

Quality  Assurance  - A system  of  activities  whose  purpose  is  to  provide  to  the 
producer  or  user  of  a product  or  a service  the  assurance  that  it 
meets  defined  standards  of  quality.  It  consists  of  two  separate  but 
related  activities,  quality  control  and  quality  assessment  (defined). 
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Quality  Circle  - A small  group  of  individuals  with  related  interests  that 
meets  at  regular  intervals  to  consider  problems  or  other  matters 
related  to  the  quality  of  outputs  of  a process  and  to  the  correction 
of  problems  or  to  the  improvement  of  quality. 

Quality  Control  - The  overall  system  of  activities  whose  purpose  is  to 

control  the  quality  of  a product  or  service  so  that  it  meets  the 
needs  of  users.  The  aim  is  to  provide  quality  that  is  satisfactory, 
adequate,  dependable,  and  economic. 

Random  Sample  - A sample  selected  from  a population,  using  a randomization 
process . 

Reduction  - The  process  of  preparing  one  or  more  subsamples  from  a sample. 

Reference  Material  (RM)  - A material  or  substance  one  or  more  properties  of 
which  are  sufficiently  well  established  to  be  used  for  the  calibra- 
tion of  an  apparatus,  the  assessment  of  a measurement  method,  or  for 
the  assignment  of  values  to  materials. 

Reference  Method  - A method  which  has  been  specified  as  capable,  by  virtue  of 
recognized  accuracy,  of  providing  primary  reference  data. 

Relative  Standard  Deviation  - The  coefficient  of  variation,  expressed  as  a 
percentage . 

Replicate  - A counterpart  of  another,  usually  referring  to  an  analytical 

sample  or  a measurement.  It  is  the  general  case  for  which  duplicate 
is  the  special  case  consisting  of  two  samples  or  measurements. 

Routine  Method  - A method  used  in  recurring  analytical  problems. 

Sample  - A portion  of  a population  or  lot.  It  may  consist  of  an  individual 
or  groups  of  individuals.  It  may  refer  to  objects,  materials,  or  to 
measurements,  conceivable  as  part  of  a larger  group  that  could  have 
been  considered. 

Secondary  Standard  - A standard  whose  value  is  based  upon  comparison  with 
some  primary  standard.  Note  that  a secondary  standard,  once  its 
value  is  established,  can  become  a primary  standard  for  some  other 
user . 

Segment  - A specifically  demarked  portion  of  a lot,  either  actual  or 
hypothetical . 

Selectivity  - The  ability  of  methodology  or  instrumentation  to  respond  to  a 
desired  substance  or  constituent  and  not  to  others.  It  is  sometimes 
quantified  as  cross  sensitivity,  which  see. 

Sensitivity  - Capability  of  methodology  or  instrumentation  to  discriminate 
between  samples  having  differing  concentrations  or  containing 
differing  amounts  of  an  analate. 

Significant  Figure  - A figure (s)  that  remains  to  a number  or  decimal  after 
the  ciphers  to  the  right  or  left  are  cancelled. 
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Special  Cause  - A cause  of  variance  or  bias  that  is  external  (not  inherent) 
to  the  measurement  system. 

Split  Sample  - A replicate  portion  or  sub-sample  of  a total  sample  obtained 

in  such  a manner  that  it  is  not  believed  to  differ  significantly  from 
other  portions  of  the  same  sample. 

Standard  - A substance  or  material,  the  properties  of  which  are  believed  to 
be  known  with  sufficient  accuracy  to  permit  its  use  to  evaluate  the 
same  property  of  another.  In  chemical  measurements,  it  often 
describes  a solution  or  substance,  commonly  prepared  by  the  analyst, 
to  establish  a calibration  curve  or  the  analytical  response  function 
of  an  instrument. 

Standard  Addition  - A method  in  which  small  increments  of  a substance  under 
measurement  are  added  to  a sample  under  test  to  establish  a response 
function  or,  by  extrapolation,  to  detemine  the  amount  of  a constitu- 
ent orginally  present  in  the  test  sample. 

Standard  Method  - A method  (or  procedure)  of  test  developed  by  a standards- 
writing  organization,  based  on  consensus  opinion  or  other  criteria, 
and  often  evaluated  for  its  reliability  by  a collaborative  testing 
procedure . 

Standard  Operations  Procedure  (SOP)  - A procedure  adopted  for  repetitive  use 
when  performing  a specific  measurement  or  sampling  operation.  It  may 
be  a standard  method  or  one  developed  by  the  user. 

Standard  Reference  Material  - A reference  material  distributed  and  certified 
by  the  National  Bureau  of  Standards. 

Strata  - Segments  of  a lot  that  may  vary  with  respect  to  the  property  under 
study. 

Subsample  - A portion  taken  from  a sample.  A laboratory  sample  may  be  a 
subsample  of  a gross  sample;  similarly,  a test  portion  may  be  a 
subsample  of  a laboratory  sample. 

Technique  - A physical  or  chemical  principle  utilized  separately  or  in 
combination  with  other  techniques  to  determine  the  composition 
(analysis)  of  materials. 

Test  Portion  (also  called  specimen,  test  specimen,  test  unit,  aliquot)  - 
That  quantity  of  a material  of  proper  size  for  measurement  of  the 
property  of  interest.  Test  portions  may  be  taken  from  the  gross 
sample  directly,  but  often  preliminary  operations,  such  as  mixing  or 
further  reduction  in  particle  size,  are  necessary. 

Tolerance  Interval  - That  range  of  values,  calculated  from  an  estimate  of  the 
mean  and  the  standard  derivation,  within  which  a specified  percentage 
of  Individual  values  of  population  (measurements  or  sample)  are 
expected  to  lie  with  a stated  level  of  confidence 
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Traceability  - The  ability  to  trace  the  source  of  uncertainty  of  a 
measurement  or  a measured  value. 


Training  - Formal  or  informal  instruction  designed  to  provide  competence  of  a 
specific  nature. 

Uncertainty  - The  range  of  values  within  which  the  true  value  is  estimated  to 
lie.  It  is  a best  estimate  of  possible  inaccuracy  due  to  both  random 
and  systematic  error. 

Validation  - The  process  by  which  a sample,  measurement  method,  or  a piece 
of  data  is  deemed  to  be  useful  for  a specified  purpose. 


Variance  - The  value  approached  by  the  average  of  the  sum  of  the  squares  of 
deviations  of  individual  measurements  from  the  limiting  mean. 
Mathematically,  it  may  be  expressed  as 


I(x^  - m)^ 
n 


as  n ^ ® 


p 

Ordinarily  it  cannot  be  known  but  only  its  estimate,  s , which  is 
calculated  by  the  expression 


2 _ - x)^ 

n-1 


Warning  Limits  - The  limits  shown  on  a control  chart  within  which  most  of  the 
test  results  are  expected  to  lie  (within  a 95%  probability)  while  the 
system  remains  in  a state  of  statistical  control. 

Youden  Plot  - A graphical  presentation  of  data,  recommended  first  by  W.  J. 

Youden,  in  which  the  result (s)  obtained  by  a laboratory  on  one  sample 
is  plotted  with  respect  to  the  result(s)  it  obtained  on  a similar 
sample.  It  helps  in  deciding  whether  discrepant  results  are  due 
to  random  or  systematic  error. 
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